Data marshalling solution for mult-tier application

Question

I am developing a multi-tier software solution where I have 1 server, n thin client engines(TCE) and m thin clients at the most basic level. I plan on having many other components for scalability where extra components can be added for congestion etc. but these are the basic components.

When I originally designed the solution I created my own command protocol for communication between the server and TCE as well as the TCE and the thin clients using JSON. I came to the development of datasets and data transfer instead of just messages. The application could easily house 100GB of data and since it is an in memory solution(database for persistence only), I need to marshal the data between different tiers as well as handle delta transfers(all on the same network but different hardware). I quickly realized that all the synchronization logic that I need to write will be immense so I decided to take a step back and put some thought into this before I get too far along and committed.

Since I am the only developer I am always looking for ways to minimize effort so I can accomplish more. I decided to transition to the Java built-in serialization since my components that house dataset versions are 100% Java. After doing this and testing, I obviously ran into a lot of errors around what can be serialized. As you may know any instance of a class that you want to place in the object stream needs to implement the Serializable interface. I realized that the graph is very large when I am trying to transfer an object. This means I need to modify all my classes that are used between tiers to be serializable and mark server only fields as transient. This is also a lot of work.

So I wanted to ask the community what they believe the best design should be here. My intention for using Java serialization was that I could assign a root object to be transferred and everything related downstream gets sent as well which makes things very easy for me. Also for delta processing this could make things much easier on me as a developer since I would not have to write nearly as much data synchronization logic.

The goals I am trying to achieve are, a solution that requires minimal modification to all the classes used between tiers and a solution that is scalable.

I looked into google's protocol buffers but that is something that suites my thin client communication better since I will have a Java Thin Client, Web Client, and Mobile client.

Answer 1

I recommend using something like ProtocolBuffers or Apache Avro, that way you can define your data simply, and your data isn't tied to a language implementation. It is a big system, you might want to implement some part separately later, or you might want to make architectural changes. Using a portable data serialization library, you can isolate your data storage more easily.

Answer 2

Other than 100 GiB, you didn't give us any specifications. In particular, do you primarily care about latency or throughput?

Here's the simplest thing that could possibly work for such vague specs: persist everything, treat that as the Source of Truth. Then pass handles around, so your (small, serializable) objects are essentially pointers to the persisted data structures. You mentioned computing deltas. You may find it convenient to think of "processing" a large object as "computing a delta" to be applied on top of that object. There's no need to "synchronize" when you're writing to an append-only object store and accumulating delta descriptors that modify your interpretation of an evolving object.

BTW, java's serialization support can be convenient for some projects, but introspection slows it down. Avro and others show the way to faster and more compact approaches.