1

In my current Java EE project I've got a quite common task: Load objects from database using Hibernate, transfer them in another object structure (based on a XSD) and serialize it using JAXB. To simplify the scenario imagine the XML I want so serialize is a file with information records to requests we get. In theory there can be infinite information records to the requests, but I estimate a pessimistic but still realistic number of records I have to serialize.

I first made a rough estimate of the resulting XML file size and got to 292MB (about 9KB per single record). I already expected that the file size will outrange the maximium heap size of 50MB a java request is allowed to gather by our operators / server admins. So I need to find a limit where to split the records into several processes and files. I know that I can't just cut down the number of records to serialize in one file by the resulting file size, meaning I can't calculate "okay x records are 200 MB then x/4 records are 50MB and that's fits with the maximum heap". For sure I have to keep the loaded objects and the object structure I serialize in mind.

But how can I roughly estimate the needed heap size by the final size a single record so I can serialize a maximum number of records without running into memory issues? As said a rough calculation is enough, like "if your final size per record is 6KB then you need 12KB (ratio 1:2) to process it in your process.

  • 1
    This will depend hugely on the garbage collection settings of your JVM and how much garbage you produce along the way (which is related to programming style). – Bart van Ingen Schenau Sep 1 '17 at 8:24
  • @BartvanIngenSchenau Do you have information about which settings are relevant and I should ask my admins? The code of the method just loads objects from DB, transfers them to the JAXB classes and serialies them. No other work done in the process. – bish Sep 1 '17 at 8:30
  • FWIW I usually serialize twice: in a first pass, I just count the bytes instead of actually writing them, then create the appropriate buffer, then do the real serialization. Of course you might not be able to do that. If the data is entirely in memory, has low granularity - small number of big objects - and/or rhe overall size is low enough, the first pass is really fast. – Tibo Sep 1 '17 at 12:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.