This is a question about solving a particular problem I am struggling with, I am parsing a long list of text data, line by line for a business app in PHP (cron script on the CLI). The file follows the format:

    HD: Some text here {text here too}

    DC: A description here
    DC: the description continues here
    DC: and it ends here.

    DT: 2012-08-01

    HD: Next header here {supplemental text}

    ... this repeats over and over for a few hundred megs

I have to read each line, parse out the HD: line and grab the text on this line. I then compare this text against data stored in a database. When a match is found, I want to then record the following DC: lines that succeed the matched HD:.

Pseudo code:

    while ( the_file_pointer_isnt_end_of_file) {
        line = getCurrentLineFromFile
        title = parseTitleFrom(line)
        matched = searchForMatchInDB(line)
        if ( matched ) {
            recordTheDCLines  // <- Best way to do this?

My problem is that because I am reading line by line, what is the best way to trigger the script to start saving DC lines, and then when they are finished save them to the database?

I have a vague idea, but have yet to properly implement it. I would love to hear the communities ideas\suggestions!

Thank you.

  • 2
    With loading and saving line by line you will have massive overhead. I would read/write in chunks. – Pieter B Dec 8 '12 at 20:49
  • I see your point. This is how I was told to do it, and so I have to implement it this way; however, thanks for the suggestion. I may work it in for optimization! – Jarrod Dec 8 '12 at 22:59
  • sed/awk to read and parse then call your php script to check the database and update if needed. – Steve Dec 9 '12 at 1:44
  • What database are you using? – NoChance Dec 9 '12 at 9:36

Separate the problem -- one script plows through and reads and stuffs the interesting stuff into some sort of data store. Second script pulls from the data store and processes the records. I suspect this will be much faster than doing it in the same script for no other reason than the 2nd script effectively multi-threads the app.


Write a two functions or a class LineReader with the following functions:

  • string GetNextLine() : reads next line from file
  • string PeekLine() : gets the next line from file, but don't move the file pointer

(you can implement this easily by a line buffer consisting of a string variable holding one line in advance; GetNextLine has to make use of that buffer as well as PeekLine).

Then, the implementation of recordDCLines should be something like

    // process line, append it to a buffer
 // here, store the found DC block

EDIT: some pseudo code, I am not experienced in PHP, but I hope you get the general idea:

 void OpenFile()
     // do stuff here to open file
     // ...
     $nextline = getNextLineFromFile();
     $endoffile = false;

 string GetNextLine()
             $nextline = getNextLineFromFile();
         $result ="";
      return $result;

 string PeekLine()
     return $nextline;
  • Thank you very much for your answer. This is a wonderful suggestion, and I will investigate it more. Could you explain this a bit more: (you can implement this easily by a line buffer consisting of a string variable holding one line in advance; GetNextLine has to make use of that buffer as well as PeekLine)? – Jarrod Dec 8 '12 at 22:57
  • @Jarrod; see my edit – Doc Brown Dec 9 '12 at 14:01

Implement a basic state machine. As you are reading lines, note the last 'command' (dc, dt, etc). When you get a 'HD', do your lookups. When you are in a DC state, you know to accumulate the message until the next item isnt a DC entry, at which point you do a write.


You could consider writing a PHP extension in C or C++ for that purpose; you could then use low-level, but efficient syscalls (e.g. mmap(2), read(2) into a large buffer, readahead(2), etc...)

You could also delegate to a helper program written in C.

  • Thanks for the suggestion. Unfortunately this is delegated work and I have to implement it as I was told! – Jarrod Dec 8 '12 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.