How to parse XML files too big to fit in memory

Question

I'm trying to process XML files that are too big to fit in memory. They range in size anywhere from dozens of megabytes to over 120GB. My initial attempt had me reading the files as plain text, in chunks of a few thousand characters at a time, and looking for individual completed XML tags in the little String chunks:

FileReader fileReader;
    try {
        fileReader = new FileReader(file);

        DocumentBuilder factory = DocumentBuilderFactory.newInstance().newDocumentBuilder();
        Document doc;

        int charsToReadAtOnce = 1000;
        char[] readingArray = new char[charsToReadAtOnce ];
        int readOffset = 0;
        StringBuilder buffer = new StringBuilder(charsToReadAtOnce * 3);

        while(fileReader.read(readingArray, readOffset, charsToReadAtOnce ) != -1) {
            buffer.append(new String(readingArray));
            String current = buffer.toString();

            doc = factory.parse(new InputSource(new StringReader(buffer.toString())));

            //see if string contains a complete XML tag
            //if so, save useful info and manually clear it
        }
    } catch (ALL THE EXCEPTIONS...

This was getting messy and complicated fast with a lot of edge cases like tags over 1000 characters long and ignoring start and end tags. Instead of pressing on, I want to use a less painful algorithm but can't come up with a real good one. Does Java have a more appropriate way to handle massive XML files like these? While asking this question, I came across Read a zipped xml with .NET. I think something like that but obviously for Java might work for me but I don't know if it exists?

Answer 1

Streaming API (such as SAX see https://docs.oracle.com/javase/tutorial/jaxp/sax/) vs DOM api's. Former one process tags as they occur, while the latter represents the entire DOM model in memory. See also https://stackoverflow.com/q/6828703/744133

Answer 2

While YoYo has provided with a good answer that works for almost any application, I have something that may be better than pure streaming parsers.

Consider, for example, a file that has a billion bank accounts:

<accounts>
    <account>
        <id>1</id>
        <balance>123.45</balance>
    </account>
    ....
</accounts>

Now, you could parse this entire file with a streaming parser. The problem, however, is that streaming parsers are cumbersome to use and tree parsers are much nicer to use. Unfortunately, due to memory constraints, you cannot represent the whole file in one big tree.

The solution is when seeing an "<account>" tag to turn on the tree collection mode and when seeing an "</account>" tag finalize the tree so that you will have the following tree whenever encountering the closing tag:

<account>
    <id>1</id>
    <balance>123.45</balance>
</account>

Then you can access the fragment of the document as a tree in a convenient manner, but the whole document is not in the memory as one big tree.

You will need to develop such tree collection code on top of a streaming parser.

Answer 3

I created a code generator designed to solve this particular problem (an early version was conceived in 2008). Basically each complexType has its Java POJO equivalent and handlers for the particular type are activated when the context changes to that element. I used this approach for SEPA, transaction banking and for instance discogs (30GB). You can specify what elements you want to process at runtime, declaratively using a propeties file.

XML2J uses mapping of complexTypes to Java POJOs on the one hand, but lets you specify events you want to listen on. E.g.

account/@process = true
account/accounts/@process = true
account/accounts/@detach = true

The essence is in the third line. The detach makes sure individual accounts are not added to the accounts list. So it won't overflow.

class AccountType {
    private List<AccountType> accounts = new ArrayList<>();

    public void addAccount(AccountType tAccount) {
        accounts.add(tAccount);
    }
    // etc.
};

In your code you need to implement the process method (by default the code generator generates an empty method:

class AccountsProcessor implements MessageProcessor {
    static private Logger logger = LoggerFactory.getLogger(AccountsProcessor.class);

    // assuming Spring data persistency here
    final String path = new ClassPathResource("spring-config.xml").getPath();
    ClassPathXmlApplicationContext context = new   ClassPathXmlApplicationContext(path);
    AccountsTypeRepo repo = context.getBean(AccountsTypeRepo.class);


    @Override
    public void process(XMLEvent evt, ComplexDataType data)
        throws ProcessorException {

        if (evt == XMLEvent.END) {
            if( data instanceof AccountType) {
                process((AccountType)data);
            }
        }
    }

    private void process(AccountType data) {
        if (logger.isInfoEnabled()) {
            // do some logging
        }
        repo.save(data);
    }
}   

Note that XMLEvent.END marks the closing tag of an element. So, when you are processing it, it is complete. If you have to relate it (using a FK) to its parent object in the database, you could process the XMLEvent.BEGIN for the parent, create a placeholder in the database and use its key to store with each of its children. In the final XMLEvent.END you would then update the parent.

Note that the code generator generates everything you need. You just have to implement that method and of course the DB glue code.

There are samples to get you started. The code generator even generates your POM files, so you can immediately after generation build your project.

The default process method is like this:

@Override
public void process(XMLEvent evt, ComplexDataType data)
    throws ProcessorException {


/*
 *  TODO Auto-generated method stub implement your own handling here.
 *  Use the runtime configuration file to determine which events are to be sent to the processor.
 */ 

    if (evt == XMLEvent.END) {
        data.print( ConsoleWriter.out );
    }
}

Downloads:

• https://github.com/lolkedijkstra/xml2j-core
• https://github.com/lolkedijkstra/xml2j-gen
• https://sourceforge.net/projects/xml2j/

First mvn clean install the core (it has to be in the local maven repo), then the generator. And don't forget to set up the environment variable XML2J_HOME as per directions in the usermanual.