4

I have a large XML file (about 75,000 lines) where I have to build a catalogue (houses) from. Building the lists works fine, but now I have a problem.

The catalogue should have a detailed presentation page of one house. One house (<item id="123">) has about 800-1200 lines of data, based on the house type.

Which is the best way, as referring to making the script faster and saving lines of code, to read these data and present them?

Some houses, for example, have a sauna, and when these data are in the XML file, the presentation page should contain a section sauna.

I tried before to read the whole XML content with a recursive function into arrays and with a lot of foreaches (maximum depth of children is three) with SimpleXML, but it was really ugly slow, and the recursion did not work at all, because my computer could not handle so much input.

Is there any other way to build this data except to query every variable with if?

  • 3
    What about moving this data into a database, either relational or (maybe better for some types of query NoSQL)? – thorsten müller Apr 17 '13 at 12:04
  • 1
    Hint: you can almost always remove the word "dynamically" from such questions. It adds nothing (try it: read the text, but skip every "dynamically"). – Joachim Sauer Apr 17 '13 at 12:45
  • Tearing through XML is one of the (few) areas that PHP excels. I've personally found the DOMDocument class to be more handy than simplexml. – CamelBlues Apr 17 '13 at 14:21
  • 2
    @JoachimSauer, don't worry, I dynamically edited the question to fix that. – user82096 Apr 18 '13 at 4:33
  • Thank you about the suggestions! English is not my native language, but now I know to avoid dynamically :) – Owl Apr 18 '13 at 11:02
9

XML is an inefficient method of storing large amounts of data. It uses a lot of disk space (look at an XML file and note what a large portion of it is taken up by the syntax and structure definition), and it is slow and memory-intensive to access. The whole tree (or at least a large portion of it) must be parsed just to get a single element, and XML parsers often use several times the size of the file in memory space to do this.

If you need to do something performance sensitive (such as load information onto a web page), 75,000 lines of XML just isn't going to be fast.

If performance matters, you should really move the information into a relational database, as suggested by thorsten müller. Then your task will become trivial. Even if you have no choice but to receive the data in XML, have your program perform a one-time load of that XML file into the database whenever it is updated, and then use the database the rest of the way. Besides being faster, the database will also be a lot easier to work with.

If you choose to stay with XML, you can get some help with your algorithm, but more information is needed. I suggest posting the portion of the code that you describe on Code Review.

  • There's no need to parse the whole tree at once when reading XML. Large XML files can be handled using event based parsing. – COME FROM Apr 18 '13 at 12:02
  • @COMEFROM, yes, that is why I added the qualification (or a large portion of it). Since the XML is stored in a text file and doesn't have random access, you have to go through a lot of the file, on average, just to get a single element. – user82096 Apr 18 '13 at 12:09
  • I have a question here, I am working with HotelsPro API which contains an XML file of 96000+ hotels around the world. I want to save them all in mysql database without facing execution time error. Is that possible anyhow? Please let me know. – William Francis Gomes Nov 2 '14 at 10:13
  • @WilliamFrancisGomes, try it yourself, and if you run into specific problems, then post a question about it on the appropriate site (probably Stackoverflow, not here). – user82096 Nov 3 '14 at 12:24
7

Maybe you're running into the limitation that SimpleXML tries to read the entire document in one step, and your document is large or complex. An alternative is to use a stream- or event-based parser which reads bits at a time, sets the state of the parser based on what was read, and lets you register handlers that will react to specific events. Here is a explanation of the idea with a simple example: http://sax.sourceforge.net/event.html

I'm not very familiar with PHP, but it looks like there is something like SAX for you to try: http://php.net/manual/en/book.xml.php. From the introduction:

This extension lets you create XML parsers and then define handlers for different XML events.

Searching for sax in php on Google also shows some promising leads.

1

Recently I posted about how to achieve that, here an overview of the post:

Over the past 4 years I have worked with many XML providers (interconnecting B2B applications) and some of these providers distribute big XML files, some of them have more than 2GB, as you may already suspect we should not dump 2GB of information into memory because PHP will generate a memory overflow fatal error and your application will not be able to recover by itself, so in this post I will explains how to solve this problem, using:

  • Generators: Generators provide an easy way to build simple iterators without implementing the Iterator interface for more information read: New features of PHP 5.5

  • XMLReader: is a PHP implementation on the shoulder of libxml that allows to read XML files effeciently since it acts as a forward cursor.

  • SimpleXMLElement: Create an XML object from an XML string.

Read XML files efficiently in PHP

1

If you are presenting the information via web pages, you can split the XML data up so that:

a. The main file contains just the introductory and meta data for ALL the houses/products/services.

b. The details of each house/product/service are in separate file for each.

The idea is that:

a. Your listing page will just use the main file.

b. Your detail page for a house/product/service will:

_ 1. Load up the main XML.

_ 2. Load and append the house/product/service-specific XML to the required node in the main XML. (Though since it is the only one, you could just append it to the root node.)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.