This may seem pedantic, but you cannot apply Agile to the letter, it is just a series of values and principles. There are probably two aspects of what you are referring to. First, there are frameworks and methodologies in the agile umbrella and there are some very strong arguments for following those to the letter (to start). Second, there is an obsession in many industries with best practices and the idea that saying what you will do and then following that no matter what is somehow a good thing (spoilers: it isn't).
Frameworks in Agile
Scrum, XP, Kanban, etc are all pre-packaged approaches to getting certain results. You don't have to use them, but they are a shortcut off of someone else's success (or really, thousands of someone elses). The catch is, you void the warranty if you don't practice it as intended. Think of it this way: I need a new shelf. I could build a shelf in my garage, but that would require that I buy a bunch of tools, learn carpentry, practice and fail many times. Instead, I'm going to IKEA and buying a shelf. Now, I get the shelf home and I want to put it together. Should I follow the instructions or just start putting screws in holes until I run out of both? Well, it's my shelf, so I don't have to follow the instructions, but if I put it together a different way and throw the stabilizing brace in the trash because it uses a flat-head screw and I only have a philips driver, can I really complain when the shelf breaks?
I hope every team that uses scrum or XP eventually grows past it, keeping the parts that work and retooling others to fit their unique context. But you do that after you've used the out-of-the-box option and really learned why it works like it does. Then you customize.
Best Practices
To borrow some phrasing from the cynefin model, best practices only work with clear, simple problem. Complicated problems have a whole bunch of "good" practices with each one being slightly better depending on your circumstances. If you get into complex problems, there are only emergent practices. Knowledge work falls almost entirely in complicated and complex spaces, meaning best practices aren't really applicable. That means no practice should be taken as obviously the right thing to do. Everything is up for review. I have no idea how organizations have become obsessed with best practices, but it's really problematic.
Now, to tie those together: If I'm using Kanban, I should be using WIP limits. Otherwise, I'm not using Kanban. A corvette with a motorcycle engine in it isn't a corvette. Does that negate my statement about best practices? No. the practice of WIP limits should be up for review. But if you've never used this before, you don't have the knowledge to effectively evaluate its effectiveness in your context. If you try using a little piece of it, it's hard to tell if it didn't work because it's a bad fit or because you didn't use the supporting practices alongside. So, you practice one of these frameworks in its entirety to learn how and why they work, then start challenging them.
