Techniques for parsing XML

Question

I've always found XML somewhat cumbersome to process. I'm not talking about implementing an XML parser: I'm talking about using an existing stream-based parser, like a SAX parser, which processes the XML node by node.

Yes, it's really easy to learn the various APIs for these parsers, but whenever I look at code that processes XML I always find it to be somewhat convoluted. The essential problem seems to be that an XML document is logically separated into individual nodes, and yet the data types and attributes are often separated from the actual data, sometimes by multiple levels of nesting. Therefore, when processing any particular node individually, a lot of extra state needs to be maintained to determine where we are and what we need to do next.

For example, given a snippet from a typical XML document:

<book>
  <title>Blah blah</title>
  <author>Blah blah</author>
  <price>15 USD</price>
</book>

...How would I determine when I've encountered a text node containing a book title? Suppose we have a simple XML parser which acts like an iterator, giving us the next node in the XML document everytime we call XMLParser.getNextNode(). I inevitably find myself writing code like the following:

boolean insideBookNode = false;
boolean insideTitleNode = false;

while (!XMLParser.finished())
{
    ....
    XMLNode n = XMLParser.getNextNode();

    if (n.type() == XMLTextNode)
    {
        if (insideBookNode && insideTitleNode)
        {
            // We have a book title, so do something with it
        }
    }
    else
    {
        if (n.type() == XMLStartTag)
        {
            if (n.name().equals("book")) insideBookNode = true
            else if (n.name().equals("title")) insideTitleNode = true;
        }
        else if (n.type() == XMLEndTag)
        {
            if (n.name().equals("book")) insideBookNode = false;
            else if (n.name().equals("title")) insideTitleNode = false;
        }
    }
}

Basically, the XML processing quickly turns into a huge, state-machine driven loop, with lots of state variables used to indicate parent nodes we've found earlier. Otherwise, a stack object needs to be maintained to keep track of all the nested tags. This quickly becomes error-prone and difficult to maintain.

Again, the problem seems to be that the data we're interested in is not directly associated with an individual node. Sure, it could be, if we wrote the XML like:

<book title="Blah blah" author="blah blah" price="15 USD" />

...but this is rarely how XML is used in reality. Mostly we have text nodes as children of parent nodes, and we need to keep track of the parent nodes in order to determine what a text node refers to.

So...am I doing something wrong? Is there a better way? At what point does using an XML stream-based parser become too cumbersome, so that a fully-fledged DOM parser becomes necessary? I'd like to hear from other programmers what sort of idioms they use when processing XML with stream-based parsers. Must stream-based XML parsing always turn into a huge state machine?

Answer 1

To me, the question is the other way round. At what point does an XML Document become so cumbersome, that you have to start using SAX instead of DOM?

I would only use SAX for a very large, indeterminately-sized stream of data; or if the behaviour the XML is intended to invoke is really event-driven, and therefore SAX-like.

The example you give looks very DOM-like to me.

1. Load the XML
2. Extract the title node(s) and "do something with them".

EDIT: I'd also use SAX for streams that may be malformed, but where I want make a best-guess at getting the data out.

Answer 2

I don't work with XML too much, bit in my opinion, probably one of the best ways of parsing XML with a library is using XPath.

Instead of traversing the tree to find some specific node, you give a path to it. In the case of your example (in pseudocode), it would be something like:

books = parent.xpath("/book") // This would give you all the book nodes
for-each book in books
    title = book.xpath("/title/text()")
    author = book.xpath("/author/text()")
    price = book.xpath("/price/text()")

    // Do things with the data

XPath is much more powerful than that, you can search using conditions (both on values and attributes), select a specific node in a list, move levels through the tree. I recommend you look for info on how to use it, it is implemented in a lot of parsing libraries (I use it the .Net Framework version and lxml for Python)

Answer 3

Must stream-based XML parsing always turn into a huge state machine?

Usually it does, yes.

For me to point to use a fully-fledged DOM parser is when I would need to mimic parts of the file hierarchy in-memory, for example to be able to resolve cross-references within the document.

Answer 4

Parsing in general is simply driving a state machine, and XML parsing is no different. Stream-based parsing is always a hassle, I always wind up building a stack of some sort to keep track of ancestor nodes, and defining a lot of events and some kind of event dispatcher that checks a tag or path registry and fires off an event if one matches. The core code is fairly tight, but I wind up with a huge wad of event handlers that mostly consist of assigning the value of the following text node to a field in a structure somewhere. It can get pretty hairy if you need to mix business logic in there too.

I would always use DOM unless size or performance issues dictated otherwise.

Answer 5

Not completely language agnostic, but I typically deserialize the XML into objects rather than even think about parsing. Only time to worry about parsing strategies per se is if you have a speed issue.

Answer 6

It becomes much less cumbersome if you can use XPath. And in .Net land LINQ to XML abstracts a lot of the less glamorous stuff too. (Edit - these require a DOM approach of course)

Fundamentally, if you are taking a stream based approach (so you can't use nicer abstractions that require a DOM) I think it will always be pretty cumbersome and I'm not sure there is any way around this.

Answer 7

If you can find a parser that gives you an iterator, have you thought of treating it as a lexer, and using a state machine generator?