There are definitely pros/cons about using JSON over REST vs. straight up TCP/IP with binary protocol and I think you are already suspecting that binary protocol will be faster. I can't tell you exactly how much faster (and this would depend on a lot of factors), but I would guess maybe 1-2 orders of magnitude difference.
At first glance if something is 10-100 times slower than something else, you might have a knee-jerk reaction and go for "fast thing". However, this speed difference is only in the protocol itself. If there's database/file access on the server side, that won't get impacted by your choice of the transfer layer. In some cases, it might make your transfer layer speed much less significant.
HTTP REST and JSON are good for a number of reasons:
- they are easily consumable by just about anyone. You can write your Web App, then turn around and publish your API for the rest of the world to use. Now anyone can hit the same end-points and get to your services
- they are easily debuggable, you can open a packet sniffer or simply dump incoming requests to text files and see what's going on. You can't do that with binary protocols
- they are easily extendable. You can add more attributes and data at a later time and not break compatibility with old clients.
Protobufs over TCP/IP:
If it was my choice, I would hands down go with HTTP REST and JSON. There's a reason that so many other companies and websites went that route. Also keep in mind that in the future you could always support 2 end points. If your design is correct, your end-point choice should be completely decoupled from your server-side business logic or the database. So if you realize later on that you need more speed for all/some requests, you should be able to add protobufs with minimal fuss. Right off the bat however, REST/JSON will get you off the ground faster and get you further.
As far as Netty vs Spring goes. I haven't used Netty directly, but I believe it is just a light-weight web server where as Spring is a framework that provides a lot more for you than just that. It has data access layers, background job scheduling and (I think) an MVC model, so it is much more heavyweight. Which one to choose? If you decided to go HTTP way, then next question is probably how standard is your app? If you are about to write some crazy custom logic that doesn't fit the standard mold and all you need is just a HTTP server layer, go with Netty.
However, I'm suspecting you app isn't that special and it could probably benefit from a lot of things that Spring has to offer. But that means that you should structure your app around Spring's framework and do things the way they expect you to do, which would mean learning more about Spring before diving into your product. Frameworks in general are great because again they get you off the ground faster, but the downside is that you have to fit into their mold instead of doing your own design and then expect the framework to just work.
(*) - in the past it was pointed out that my posts do not reflect opinions of the entire world, so I'll go on the record and just add that I have limited experience with either Netty (I've used Play framework before which is based on Netty) or Spring (I've only read about it). So take what I say with a grain of salt.