I have a requirement to write a POST endpoint where the payload is huge(not normal). In the order 10s of Megabytes. And this endpoint will be accessed by a lot of clients (in the order of 1000s) simultaneously.

How should I go about designing this endpoint?

The expectation is to validate the incoming requests and send appropriate responses to the clients. I understand that serializing the entire JSON into an object tree wont scale and I am looking for something like this.

Are there other options?

  • 2
    Gson streaming API sites.google.com/site/gson/streaming – AmazingDreams Jun 24 '16 at 15:02
  • 1
    is JSON the best/only possible format for this data? – Ewan Jun 24 '16 at 15:04
  • A majority of the clients already deal with JSON but if there are clear performance gains, I could convince them. Or, worst case write a converter. – Karthick R Jun 24 '16 at 15:18
  • Is there any option push back on the design? Generally large JSON payloads are built up from many smaller elements. Something like a POST of the top level and many PUTs or POSTs to the generated resource would be preferable. If you are stuck with this, are you going to have store this and if so, what does that look like? – JimmyJames Jun 24 '16 at 15:44

I don't know what kind of objects you are serializing, but generally speaking, a binary serialization, combined with something like LZ4 compression, makes much more sense than JSON.

|improve this answer|||||

Jackson and Gson both provide streaming APIs. Implement handling of request and/or building of response in terms of streams or events and you won't need to keep the whole tree in memory.

Do not try to compress data, instead ensure Jersey is doing this for you, by implementing standard HTTP compression.

Large JSON is no different from any other format for storing datum of substantial size.

|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.