We are starting to build a new app and I would like to explore the idea of doing a thick JS client (backbone / angular) with only RESTful API exposed by our application layer. What are some of the best practices and potential pitfalls for doing this kind of web app?
If you are going to have only a REST API (and no actual server-side "pages"), the biggest challenges that come to mind for me are:
It's sad, but most screen readers are way behind the times and still can't cope with dynamic content. The modern solution is WAI-ARIA, and it's a steep learning curve.
Not every workflow is going to be cleanly encapsulated by a single resource. You might need to update an /invoice and /account at the same time. There still aren't any real standards for this; the typical answer seems to be to encapsulate the transaction itself as a resource, and handle atomicity in the REST API for the transaction resource. A shopping cart is a good example of a transaction; once you checkout, the entire basket is "committed" at once.
$.ajaxor whatever, but even that is fraught with problems, since some error codes have semantic meaning (404, 409, etc.) Assuming you're even willing to go to the effort of setting up a logging API to collect errors, you'll have to make a trade-off between verbosity and bandwidth, and also put some kind of security or rate limiting on it so that people can't just spam your logging API to DoS your site.
At least in my experience, testing becomes more difficult because you need to maintain parallel suites of unit tests for client and server. If you do browser-based integration testing, it will be harder to tell if the failures are coming from the client or server side, unless you also maintain another suite of API-only integration tests. Basically, it's like adding another "tier".
There are standards for this (e.g. OAuth), but they're a pain to implement compared to more traditional models. This is especially true if you're running a mixed HTTP/HTTPS site and have to deal with crap like CORS. It's very much a solvable problem and lowest on my list for a reason; it'll just be extra work and bite you at odd times, because many of the back-end frameworks can be a bit... imperious about how authentication is supposed to work, and over-utilize things like session state.
Also, your app is basically spilling its guts to everyone who uses it, making it trivially easy to reverse-engineer the API, so you'd better make sure you have lots of validation in place, otherwise any random script kiddie can 0wn you. Not that this is a new problem, you're just making it a little easier for them by providing a direct line.
All that being said, these are not even close to being showstoppers. Aside from the vastly improved user experience, you gain huge benefits from this model through reusability and separation of concerns. It becomes much harder (and much less necessary) for developers to abuse session state or other temporary data. And you can truly have front-end and back-end specialists, rather than having to expose your back-end guys to markup or your front-end guys to controllers.
I very much prefer this architecture but do be aware of the trade-offs you're making. It's still kind of a new/immature field - and people are still making a lot of mistakes. I wouldn't trust any "best practices" guide this early in the game.