1

Description
I am working on an application using Angular (front-end) and Hibernate (back-end). My use case involves uploading multiple images, which I convert to base64 format and send as JSON with additional information for each image from the front-end.

The issue I'm encountering is that the HTTP request exceeds the maximum size limit due to the large base64-encoded images. I'm struggling to find an optimal solution for this problem.

What I've Tried\

  • Base64 Conversion: I have converted the images to base64 to include them in the JSON payload, which is causing the size issue.

  • Single Transaction Requirement: I need to keep the images in a single request because multiple database transactions are performed on the back-end that require all the information to be received together.

Research and Analysis \

  • Chunked Uploads: I've considered splitting the images into smaller chunks, but I am unsure how to handle the reassembly and ensure all the information is processed correctly in a single transaction.
  • Alternative File Upload Methods: I have read about multipart uploads and other methods, but I am concerned about maintaining the integrity of the database transactions and ensuring that all images and their associated data are processed together.

Request for Help
Has anyone encountered a similar issue or can suggest a solution for handling large file uploads in Angular and Hibernate without exceeding the HTTP request size limit? Any guidance or best practices would be greatly appreciated.

Thank you!

4
  • 1
    how large are we talking? pre base64 conversion
    – Ewan
    Commented Jun 23 at 20:59
  • 1
    Why would multipart uploads cause you any issues with database integrity? It's still a single request, just one that can include a binary stream (the image) separately to the JSON stream. Commented Jun 24 at 6:31
  • How does multipart upload help? Is not it still a single request? A sum of parts is still a subject for the same limit?
    – Basilevs
    Commented Jun 25 at 6:28

2 Answers 2

6

HTTP requests most definitely can be configured on the server side to be limitless. However that won't help you much because JSON cannot be processed as a stream, or at least it is hard. Typically JSON parsers read entire buffer into memory, and then parse it all at once. Which is not a problem when payload is relatively small, but a huge one for embedded images. You really don't want to waste potentially hundreds of megabytes of memory per request (loaded buffer + parsed JSON + base64 decoding, all potentially existing at the same time).

Unless you manage to parse JSON chunk by chunk, and also base64 decode chunk by chunk. Which technically is doable, but I don't think it is an easy solution. It potentially requires a custom parser, plus it still is inefficent and has other drawbacks, as I will explain next.

I suggest you leave limits as they are and:

  1. Do not send binary files base64 encoded, and even worse as JSON. This is very inefficient, especially for big files, and even more for multiple files in a single request. In fact this is inefficient on both client and server, just don't.
  2. Drop the idea of everything being processed as a single transaction. You don't need it. Noone needs it. Especially when you deal with such big amount of data.

Honestly, I'm not even sure what "transactional" means here. Are you actually pushing the binary content into database? Not a good idea, transactions should be as fast and as small as possible. With such big content, you may kill the database. On the other hand if you write those file somewhere else, then you don't have transactional processing anyway.

The solution is quite simple. Have an upload service, some kind of CDN (which can be your own service), where you upload files as they are: raw binary data. This is the most efficient way to do that, in particular you can stream that data straight into disk with little memory overhead. Your main service will generate upload URLs where your user uploads files. These URLs can have TTLs and be signed, so that no unauthorized upload happens. Note, that if you are doing multiple uploads then this allows you to do that concurrently.

And once you upload those files, get their metadata, some kind of ids generated by the service. This can be the resource path, it would be wise to make it hard to guess, e.g. include some randomly generated chars in it. And then do the final POST with those ids as JSON. The service then processes those files in a transactional way.

So only the final step is transactional. Not the entire process. And therefore you need one final component: a background, periodic cleaner that deletes all incorrect files. And of course for it to work, you need a way of distinguishing processed files from garbage. Typically you would put some "uploaded at" timestamp, a "processed" flag, and then simply say "all files older than say 1 day and still not processed are garbage".

This can also be implemented in a more fancy way: you keep an isolated, temporary upload folder. And once you run your transaction, you move those files to final folder. The "move" operation should be fast, and independent on the size of file, i.e. only a metadata update. Which is something that all major cloud providers support out of the box. And you should support it as well, if you plan on writing your own CDN service. The benefit is that you don't have to worry about "processed" flag, where to keep it. In particular you don't need an additional database. The drawback is that the "move" operation is unlikely to be part of the main transaction. Meaning you may end up in an inconsistent state and some additional detection mechanism might be necessary. Or you can ignore it, such situations are extremely rare anyway. Your cleaner then scans only the temporary folder for old enough files. And that's it.

7
  • Why can't JSON be processed as a stream? There are dedicated adaptations and streaming parsers.
    – Basilevs
    Commented Jun 25 at 6:29
  • @Basilevs JSON is a recursive structure. Typical parsers will map it entirely into target objects. But even if you plug into parsing process, then at any given step you need to at least keep track of upper levels of JSON structure. Yeah, it might be somewhat doable if JSON is flat. But that's not really streaming.
    – freakish
    Commented Jun 25 at 7:10
  • SAX XML parsers around are very embarrassed right now.
    – Basilevs
    Commented Jun 25 at 8:35
  • @Basilevs XML has exactly the same problem. The deeper it is, the more state a parser has to keep around. In other words used memory is proportional at least to nesting level. No, this is not real streaming. You can pretend as much as you want, it doesn't change anything.
    – freakish
    Commented Jun 25 at 8:59
  • 1
    Used memory is indeed proportional to nesting level. And that's always small. Streaming allows to eliminate sequence buffering, not nesting overhead.
    – Basilevs
    Commented Jun 25 at 9:37
0

The naive approach is to increase the limit on the server. For example in if your server is powered by Nginx you can set client_max_body_size.

A more secure and resilient approach would be set up CDN endpoints like S3. Usually, different services are dedicated only to distributing (and accepting) data.

The third option if you don't want to change anything is to treat your JSON as a set of data and send parts of the JSON separately and re-built full JSON on the server side. Another advantage of such an approach is that - depending on your needs - you can save more information in the future onto JSON without sending the whole file again like it is NoSQL database.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.