I am developing a multi-tier software solution where I have 1 server, n thin client engines(TCE) and m thin clients at the most basic level. I plan on having many other components for scalability where extra components can be added for congestion etc. but these are the basic components.
When I originally designed the solution I created my own command protocol for communication between the server and TCE as well as the TCE and the thin clients using JSON. I came to the development of datasets and data transfer instead of just messages. The application could easily house 100GB of data and since it is an in memory solution(database for persistence only), I need to marshal the data between different tiers as well as handle delta transfers(all on the same network but different hardware). I quickly realized that all the synchronization logic that I need to write will be immense so I decided to take a step back and put some thought into this before I get too far along and committed.
Since I am the only developer I am always looking for ways to minimize effort so I can accomplish more. I decided to transition to the Java built-in serialization since my components that house dataset versions are 100% Java. After doing this and testing, I obviously ran into a lot of errors around what can be serialized. As you may know any instance of a class that you want to place in the object stream needs to implement the Serializable interface. I realized that the graph is very large when I am trying to transfer an object. This means I need to modify all my classes that are used between tiers to be serializable and mark server only fields as transient. This is also a lot of work.
So I wanted to ask the community what they believe the best design should be here. My intention for using Java serialization was that I could assign a root object to be transferred and everything related downstream gets sent as well which makes things very easy for me. Also for delta processing this could make things much easier on me as a developer since I would not have to write nearly as much data synchronization logic.
The goals I am trying to achieve are, a solution that requires minimal modification to all the classes used between tiers and a solution that is scalable.
I looked into google's protocol buffers but that is something that suites my thin client communication better since I will have a Java Thin Client, Web Client, and Mobile client.