11

I've always found XML somewhat cumbersome to process. I'm not talking about implementing an XML parser: I'm talking about using an existing stream-based parser, like a SAX parser, which processes the XML node by node.

Yes, it's really easy to learn the various APIs for these parsers, but whenever I look at code that processes XML I always find it to be somewhat convoluted. The essential problem seems to be that an XML document is logically separated into individual nodes, and yet the data types and attributes are often separated from the actual data, sometimes by multiple levels of nesting. Therefore, when processing any particular node individually, a lot of extra state needs to be maintained to determine where we are and what we need to do next.

For example, given a snippet from a typical XML document:

<book>
  <title>Blah blah</title>
  <author>Blah blah</author>
  <price>15 USD</price>
</book>

...How would I determine when I've encountered a text node containing a book title? Suppose we have a simple XML parser which acts like an iterator, giving us the next node in the XML document everytime we call XMLParser.getNextNode(). I inevitably find myself writing code like the following:

boolean insideBookNode = false;
boolean insideTitleNode = false;

while (!XMLParser.finished())
{
    ....
    XMLNode n = XMLParser.getNextNode();

    if (n.type() == XMLTextNode)
    {
        if (insideBookNode && insideTitleNode)
        {
            // We have a book title, so do something with it
        }
    }
    else
    {
        if (n.type() == XMLStartTag)
        {
            if (n.name().equals("book")) insideBookNode = true
            else if (n.name().equals("title")) insideTitleNode = true;
        }
        else if (n.type() == XMLEndTag)
        {
            if (n.name().equals("book")) insideBookNode = false;
            else if (n.name().equals("title")) insideTitleNode = false;
        }
    }
}

Basically, the XML processing quickly turns into a huge, state-machine driven loop, with lots of state variables used to indicate parent nodes we've found earlier. Otherwise, a stack object needs to be maintained to keep track of all the nested tags. This quickly becomes error-prone and difficult to maintain.

Again, the problem seems to be that the data we're interested in is not directly associated with an individual node. Sure, it could be, if we wrote the XML like:

<book title="Blah blah" author="blah blah" price="15 USD" />

...but this is rarely how XML is used in reality. Mostly we have text nodes as children of parent nodes, and we need to keep track of the parent nodes in order to determine what a text node refers to.

So...am I doing something wrong? Is there a better way? At what point does using an XML stream-based parser become too cumbersome, so that a fully-fledged DOM parser becomes necessary? I'd like to hear from other programmers what sort of idioms they use when processing XML with stream-based parsers. Must stream-based XML parsing always turn into a huge state machine?

2
  • 2
    if your using a .net language, you should look at linq to xml aka XLinq.
    – Muad'Dib
    Apr 8, 2011 at 14:59
  • Thank you, I thought I was the only one with this problem. Frankly, I often find the whole XML format to be more of a hindrance than a help. Yes, it allows one to store a lot of structured data in a small text file. But if you then need 20+ classes to unpack and make sense of the thing -- with no guarantee that you're not overlooking something more or less important. It's like the bunny in Monty Python's Holy Grail. Jun 3, 2016 at 15:23

7 Answers 7

9

To me, the question is the other way round. At what point does an XML Document become so cumbersome, that you have to start using SAX instead of DOM?

I would only use SAX for a very large, indeterminately-sized stream of data; or if the behaviour the XML is intended to invoke is really event-driven, and therefore SAX-like.

The example you give looks very DOM-like to me.

  1. Load the XML
  2. Extract the title node(s) and "do something with them".

EDIT: I'd also use SAX for streams that may be malformed, but where I want make a best-guess at getting the data out.

8
  • 2
    I think this is a good point. If you're parsing documents that are too big for DOM then you need to consider whether you're parsing documents that are too big for XML Apr 8, 2011 at 15:15
  • 1
    +1: Given the option, I'd always go with DOM. Unfortunately, it seems our design requirements always include "ability to handle any size document" and "must be performant", which pretty much rule out DOM-based solutions.
    – TMN
    Apr 8, 2011 at 15:29
  • 3
    @TMN, in an ideal world that requirements would rule out XML in the first place.
    – SK-logic
    Apr 8, 2011 at 15:49
  • 1
    @TMN, that sounds like one of those phantom requirements: "Of course all of our documents are only about 100KB, and the biggest we've seen is 1MB, but you never know what the future holds, so we should keep our options open and build for infinitely big documents" Apr 8, 2011 at 15:56
  • @Paul Butcher, you never know. I mean, a dump of Wikipedia is like 30GB of XML.
    – Channel72
    Apr 8, 2011 at 16:10
7

I don't work with XML too much, bit in my opinion, probably one of the best ways of parsing XML with a library is using XPath.

Instead of traversing the tree to find some specific node, you give a path to it. In the case of your example (in pseudocode), it would be something like:

books = parent.xpath("/book") // This would give you all the book nodes
for-each book in books
    title = book.xpath("/title/text()")
    author = book.xpath("/author/text()")
    price = book.xpath("/price/text()")

    // Do things with the data

XPath is much more powerful than that, you can search using conditions (both on values and attributes), select a specific node in a list, move levels through the tree. I recommend you look for info on how to use it, it is implemented in a lot of parsing libraries (I use it the .Net Framework version and lxml for Python)

1
  • That's fine if you can know and trust in advance the way in which the xml is structured. If you don't know whether, say, the width of an element will be specified as an attribute of a node or as an attribute node inside the size node of an element, then XPath is not going to be much help. Jun 3, 2016 at 15:27
5

Must stream-based XML parsing always turn into a huge state machine?

Usually it does, yes.

For me to point to use a fully-fledged DOM parser is when I would need to mimic parts of the file hierarchy in-memory, for example to be able to resolve cross-references within the document.

2
  • +1: Start with DOM. Avoid SAX.
    – S.Lott
    Apr 8, 2011 at 15:19
  • or with vtd-xml Oct 20, 2016 at 9:41
4

Parsing in general is simply driving a state machine, and XML parsing is no different. Stream-based parsing is always a hassle, I always wind up building a stack of some sort to keep track of ancestor nodes, and defining a lot of events and some kind of event dispatcher that checks a tag or path registry and fires off an event if one matches. The core code is fairly tight, but I wind up with a huge wad of event handlers that mostly consist of assigning the value of the following text node to a field in a structure somewhere. It can get pretty hairy if you need to mix business logic in there too.

I would always use DOM unless size or performance issues dictated otherwise.

1

Not completely language agnostic, but I typically deserialize the XML into objects rather than even think about parsing. Only time to worry about parsing strategies per se is if you have a speed issue.

3
  • That falls under parsing. Unless the XML in question is the output of object serialization and you have a ready-built deserialization library. But then this question doesn't appear.
    – user7043
    Apr 8, 2011 at 15:03
  • Many languages/stacks do have ready built deserialization libraries. Apr 8, 2011 at 16:54
  • Yeah, so what? My points still hold - not all XML files in the wild come in such a format, and if you have one that does, you don't ask this question as you just use that deserialization library and don't parse anything on your own, from streams or otherwise.
    – user7043
    Apr 8, 2011 at 18:26
0

It becomes much less cumbersome if you can use XPath. And in .Net land LINQ to XML abstracts a lot of the less glamorous stuff too. (Edit - these require a DOM approach of course)

Fundamentally, if you are taking a stream based approach (so you can't use nicer abstractions that require a DOM) I think it will always be pretty cumbersome and I'm not sure there is any way around this.

2
  • If you're using XPath, you're using DOM (unless you're using it with a home-grown XPath evaluator).
    – TMN
    Apr 8, 2011 at 15:25
  • yes, hence my comment about the abstractions requiring DOM... but I'll clarify, thanks!
    – Steve
    Apr 8, 2011 at 15:40
0

If you can find a parser that gives you an iterator, have you thought of treating it as a lexer, and using a state machine generator?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.