What approach to use to query multiple XML files?

Question

I'll be receiving tens of thousands of XML files, which will continue to grow with new files over time. I need to be able to query these files as a whole (like SQL querying all the records in a database). What technologies and/or approaches should I be looking at for this problem?

I feel like I should be looking into XQuery. I'm not that familiar with it, though.

UPDATE 1:

Example queries I might run:

• Return all the product code, product name, product description from all XML files in /folder 1/ where the supplier id = x and store id = y
• Return all the product code and product price from all XML files in /folder 2/ where the supplier id = x and store id = y
• Maybe join the results of the two queries above, joining on product code
• As there may be multiple results per product code for the two queries above, only get the most recent values based on some tag / attribute in the XML

UPDATE 2:

It seems like this article suggests that XQuery can be used: http://www.ibm.com/developerworks/xml/library/x-xquerymaven/index.html?ca=. As I am implementing a LAMP solution, I think there might be XQuery libraries available to PHP. Does this mean that I don't need an XML database to query loads of XML files?

Answer 1

If the nature of all your queries allows sequential processing of your XML files, you can do this without an XML database, using either xpath/xslt or XQuery. For example:

• Return all the product code, product name, product description from all XML files in /folder 1/ where the supplier id = x and store id = y
• Return all the product code and product price from all XML files in /folder 2/ where the supplier id = x and store id = y

These are a very simple xpath expression for any decent xslt processor. You just have to loop over the specific files, load each of them into memory, run an xpath query and collect the results from each file (which xpath does not for you). XQuery seems to allow to collect the results, but see below.

• Maybe join the results of the two queries above, joining on product code

This is not easily possible with xpath only (at least not very efficient). From the article you linked to, one can see an example how a join is done by XQuery - without a database - but it needs all XML files to be loaded into main memory (which is quite unrealistic when you have ten thousands of them). If you got all results from the two first two queries, however, you can "join" them by using your host language (in your case PHP), but you have to "program it manually". A decent XML database however should be capable of solving these problem for you - without loading all files into memory, by proper indexing.

So this all boils down to the question how many queries you expect, how many of them allow sequential processing, and how many don't. If you expect less than 5 different queries, I would probably not use a database for it but try to program every non-sequential requirement manually. If you expect more than 100 different queries, and you cannot even foresee how they will look like, then the use of a database is probably appropriate (but YMMV). In the end you have to find out for yourself what suits your needs best.

Answer 2

I have used eXist-db for storing and querying many XML files in different collections using xquery (http://exist-db.org). There are many ways to interface with it and it has proven to be reliable for my applications.