In my current Java EE project I've got a quite common task: Load objects from database using Hibernate, transfer them in another object structure (based on a XSD) and serialize it using JAXB. To simplify the scenario imagine the XML I want so serialize is a file with information records to requests we get. In theory there can be infinite information records to the requests, but I estimate a pessimistic but still realistic number of records I have to serialize.

I first made a rough estimate of the resulting XML file size and got to 292MB (about 9KB per single record). I already expected that the file size will outrange the maximium heap size of 50MB a java request is allowed to gather by our operators / server admins. So I need to find a limit where to split the records into several processes and files. I know that I can't just cut down the number of records to serialize in one file by the resulting file size, meaning I can't calculate "okay x records are 200 MB then x/4 records are 50MB and that's fits with the maximum heap". For sure I have to keep the loaded objects and the object structure I serialize in mind.

But how can I roughly estimate the needed heap size by the final size a single record so I can serialize a maximum number of records without running into memory issues? As said a rough calculation is enough, like "if your final size per record is 6KB then you need 12KB (ratio 1:2) to process it in your process.

  • 1
    This will depend hugely on the garbage collection settings of your JVM and how much garbage you produce along the way (which is related to programming style). Commented Sep 1, 2017 at 8:24
  • @BartvanIngenSchenau Do you have information about which settings are relevant and I should ask my admins? The code of the method just loads objects from DB, transfers them to the JAXB classes and serialies them. No other work done in the process.
    – bish
    Commented Sep 1, 2017 at 8:30
  • FWIW I usually serialize twice: in a first pass, I just count the bytes instead of actually writing them, then create the appropriate buffer, then do the real serialization. Of course you might not be able to do that. If the data is entirely in memory, has low granularity - small number of big objects - and/or rhe overall size is low enough, the first pass is really fast.
    – user44761
    Commented Sep 1, 2017 at 12:15


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.