6

I'm trying to process XML files that are too big to fit in memory. They range in size anywhere from dozens of megabytes to over 120GB. My initial attempt had me reading the files as plain text, in chunks of a few thousand characters at a time, and looking for individual completed XML tags in the little String chunks:

FileReader fileReader;
    try {
        fileReader = new FileReader(file);

        DocumentBuilder factory = DocumentBuilderFactory.newInstance().newDocumentBuilder();
        Document doc;

        int charsToReadAtOnce = 1000;
        char[] readingArray = new char[charsToReadAtOnce ];
        int readOffset = 0;
        StringBuilder buffer = new StringBuilder(charsToReadAtOnce * 3);

        while(fileReader.read(readingArray, readOffset, charsToReadAtOnce ) != -1) {
            buffer.append(new String(readingArray));
            String current = buffer.toString();

            doc = factory.parse(new InputSource(new StringReader(buffer.toString())));

            //see if string contains a complete XML tag
            //if so, save useful info and manually clear it
        }
    } catch (ALL THE EXCEPTIONS...

This was getting messy and complicated fast with a lot of edge cases like tags over 1000 characters long and ignoring start and end tags. Instead of pressing on, I want to use a less painful algorithm but can't come up with a real good one. Does Java have a more appropriate way to handle massive XML files like these? While asking this question, I came across Read a zipped xml with .NET. I think something like that but obviously for Java might work for me but I don't know if it exists?

  • I almost didn't ask this here but after reading the help center three times I think this counts as a design/architecture sort of question that doesn't ask for actual code in the answer. Please let me know if it would be better on SO. – ObvEng Mar 12 '17 at 2:07
  • Have not reached solution yet but looks like this may be helpful: andreas.haufler.info/2012/01/…. Open source library for parsing parts of XML documents at a time, designed for similar situations to this. – ObvEng Mar 12 '17 at 2:50
  • Parsing the XML is the easy part - you can do this with SAX or StAX. The tricky part is processing the information delivered by the parser - and to help you with that, we need to know what you want to do with the data. – Michael Kay Mar 12 '17 at 18:10
  • Could you provide me with a sample and an XSD? – dexter Apr 28 '18 at 18:41
11

Streaming API (such as SAX see https://docs.oracle.com/javase/tutorial/jaxp/sax/) vs DOM api's. Former one process tags as they occur, while the latter represents the entire DOM model in memory. See also https://stackoverflow.com/q/6828703/744133

| improve this answer | |
3

While YoYo has provided with a good answer that works for almost any application, I have something that may be better than pure streaming parsers.

Consider, for example, a file that has a billion bank accounts:

<accounts>
    <account>
        <id>1</id>
        <balance>123.45</balance>
    </account>
    ....
</accounts>

Now, you could parse this entire file with a streaming parser. The problem, however, is that streaming parsers are cumbersome to use and tree parsers are much nicer to use. Unfortunately, due to memory constraints, you cannot represent the whole file in one big tree.

The solution is when seeing an "<account>" tag to turn on the tree collection mode and when seeing an "</account>" tag finalize the tree so that you will have the following tree whenever encountering the closing tag:

<account>
    <id>1</id>
    <balance>123.45</balance>
</account>

Then you can access the fragment of the document as a tree in a convenient manner, but the whole document is not in the memory as one big tree.

You will need to develop such tree collection code on top of a streaming parser.

| improve this answer | |
  • 1
    "when seeing an <account> tag" - you are in essence still describing an event triggered parsing. Also for searching for a tag you still want to use a descent SAX based parser, just to handle those oddball situations - where maybe you have more than one kind of 'account'. SAX offers you good context tracking. I am curious though how easy it would be to switch live between the 2 - sounds like something I want to do. – YoYo Mar 13 '17 at 0:53
1

I created a code generator designed to solve this particular problem (an early version was conceived in 2008). Basically each complexType has its Java POJO equivalent and handlers for the particular type are activated when the context changes to that element. I used this approach for SEPA, transaction banking and for instance discogs (30GB). You can specify what elements you want to process at runtime, declaratively using a propeties file.

XML2J uses mapping of complexTypes to Java POJOs on the one hand, but lets you specify events you want to listen on. E.g.

account/@process = true
account/accounts/@process = true
account/accounts/@detach = true

The essence is in the third line. The detach makes sure individual accounts are not added to the accounts list. So it won't overflow.

class AccountType {
    private List<AccountType> accounts = new ArrayList<>();

    public void addAccount(AccountType tAccount) {
        accounts.add(tAccount);
    }
    // etc.
};

In your code you need to implement the process method (by default the code generator generates an empty method:

class AccountsProcessor implements MessageProcessor {
    static private Logger logger = LoggerFactory.getLogger(AccountsProcessor.class);

    // assuming Spring data persistency here
    final String path = new ClassPathResource("spring-config.xml").getPath();
    ClassPathXmlApplicationContext context = new   ClassPathXmlApplicationContext(path);
    AccountsTypeRepo repo = context.getBean(AccountsTypeRepo.class);


    @Override
    public void process(XMLEvent evt, ComplexDataType data)
        throws ProcessorException {

        if (evt == XMLEvent.END) {
            if( data instanceof AccountType) {
                process((AccountType)data);
            }
        }
    }

    private void process(AccountType data) {
        if (logger.isInfoEnabled()) {
            // do some logging
        }
        repo.save(data);
    }
}   

Note that XMLEvent.END marks the closing tag of an element. So, when you are processing it, it is complete. If you have to relate it (using a FK) to its parent object in the database, you could process the XMLEvent.BEGIN for the parent, create a placeholder in the database and use its key to store with each of its children. In the final XMLEvent.END you would then update the parent.

Note that the code generator generates everything you need. You just have to implement that method and of course the DB glue code.

There are samples to get you started. The code generator even generates your POM files, so you can immediately after generation build your project.

The default process method is like this:

@Override
public void process(XMLEvent evt, ComplexDataType data)
    throws ProcessorException {


/*
 *  TODO Auto-generated method stub implement your own handling here.
 *  Use the runtime configuration file to determine which events are to be sent to the processor.
 */ 

    if (evt == XMLEvent.END) {
        data.print( ConsoleWriter.out );
    }
}

Downloads:

First mvn clean install the core (it has to be in the local maven repo), then the generator. And don't forget to set up the environment variable XML2J_HOME as per directions in the usermanual.

| improve this answer | |
  • So xml2j streams to POJO objects keeping only a single (or few) in memory. It is an implementation of the hybrid SAX&DOM parser as suggested by @juhist, but instead of small DOM trees it returns POJOs (which are sometimes even easier to work with from within Java). Is this assumption correct? – Kasper van den Berg Apr 29 '18 at 15:41
  • 1
    Actually it never switches to tree mode. Instead on every context change it switches handlers. Each handler knows how to process that particular element type. The handler will then invoke the processor for the element if (and only if) the property @process = true is set for that element. This approach was suggested by Robert Hustead as explained here: slideshare.net/lolke/… . The article is a bit outdated, but the design still stands. I explain things in a bit more detail here: youtube.com/watch?v=QhxuKKpIJKw – dexter Apr 29 '18 at 15:57
  • @Kasper You are right. The only object that is kept in memory is the one in scope. – dexter May 2 '18 at 19:02

Not the answer you're looking for? Browse other questions tagged or ask your own question.