0

Note: this is a follow-up to this question.

I am developing a website that allows users to upload large chunks of data to my Java backend. Once validated, the data must be stored on disk. Here, "large" means in the order of 10 KB up to 100 KB.

Given that a large number of users can submit large chunks of data at the same time, and that the data must first be analyzed (validated) before being stored on disk on the server side, what options do I have to prevent JVM out-of-memory errors while processing the data chunks? Note that each data chunk is received at the moment from the frontend in a String object.

4
  • Do you have the ability to increase the amount of RAM?
    – DFord
    Commented Dec 28, 2017 at 15:45
  • 2
    You have lots of options, but your question doesn't say much about what the requirements are for processing the data. Why must it be kept in RAM? How long do you have to process it?
    – Blrfl
    Commented Dec 28, 2017 at 16:15
  • When you write/store to disk, are you writing to the file system or to a database? i.e. is a database a part of this architecture? If not, can it be?
    – tale852150
    Commented Dec 28, 2017 at 16:19
  • 4
    Why do you care if it's validated before writing to disk? Why not write it then validate it? If it doesn't pass validation, delete it.
    – Phil N DeBlanc
    Commented Dec 28, 2017 at 16:22

2 Answers 2

Reset to default
2

Write to a temporary file first

Having a server crash because you are running out of ram is an avoidable problem. Writing to a temp file allows you to get all the chunks in and analyze at once, or analyze each chunk when it comes in.

Make sure the file is deleted when you are done with your evaluations. If it passes muster, you can move the file to its permanent location with its correct filename. A filesystem move is usually just rewriting a pointer on disk so it's incredibly fast.

2
  • 1
    Thank you. Then I have one final follow up question: how do I check that the data size is less than a given number of bytes, before writing to disk? I want to avoid situations in which someone would dump hundreds of megs into the API request...
    – PleaseEditUsername
    Commented Dec 29, 2017 at 8:12
  • 1
    There's no reliable way. Check for the optional Content-Length request header. If it is present, it will tell you how many bytes the content is. If it isn't present then you will have to read in as many bytes as you can and then send a reject response if you get to your limit and clean up the temp file.
    – Berin Loritsch
    Commented Dec 29, 2017 at 13:01
1

I would echo @Berin Loritsch concerns about using memory for the same reasons mentioned.

However, I would store the data in a database and do my analysis from there.

I am probably going to start a firestorm debate (unintended) but generally speaking, databases handle sorting, searching, auditing, logging, error handling, data manipulation, security, storing, etc. better (faster, easier, more secure) than the file system, at least in most cases. Databases offer an easy means to relate and retrieve stored data using SQL.

You can use temporary tables for your operations and commit the (valid) data to the database after your analysis phase.

2
  • That really depends on your database, the analysis being performed, and how your database stores blobs. Some databases have taken measures to not store the data in the database but still let you manage it in the database (I.e. SQL Server FILESTREAM). This sounds like a golden hammer solution. It ignores solutions like cloud blob stores (like Amazon S3), or other options that can be more appropriate based on a complete solution.
    – Berin Loritsch
    Commented Dec 29, 2017 at 13:58
  • @BerinLoritsch - point well taken. Absent any detailed information about what type of analysis is being done from the OP, I had to offer a generalized solution.
    – tale852150
    Commented Dec 29, 2017 at 14:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.