9

The Senario:

You have a file with a string (average sentence worth) on each line. For arguments sake lets say this file is 1Mb in size (thousands of lines).

You have a script that reads the file, changes some of the strings within the document (not just appending but also removing and modifying some lines) and then overwrites all the data with the new data.

The Questions:

  1. Does 'the server' PHP, OS or httpd etc. already have systems in place to stop issues like this (reading/writing half way through a write)?

  2. If it does, please explain how it works and give examples or links to relevant documentation.

  3. If not, are there things I can enable or set-up, such as locking a file until a write is completed and making all other reads and/or writes fail until the previous script has finished writing?

My Assumptions and Other Information:

  1. The server in question is running PHP and Apache or Lighttpd.

  2. If the script is called by one user and is halfway through writing to the file and another user reads the file at that exact moment. The user who reads it will not get the full document, as it hasn't been written yet. (If this assumption is wrong please correct me)

  3. I'm only concerned with PHP writing and reading to a text file, and in particular, the functions "fopen"/"fwrite" and mainly "file_put_contents". I have looked at the "file_put_contents" documentation but have not found the level of detail or a good explanation of what the "LOCK_EX" flag is or does.

  4. The scenario is an example of a worst case scenario where I would assume these issues are more likely to occur, due to the large size of the file and the way the data is edited. I want to learn more about these issues and don't want or need answers or comments such as "use mysql" or "why are you doing that" because I'm not doing that, I just want to learn about file read/writing with PHP and don't seem to be looking in the right places/documentation and yes I understand PHP is not the perfect language for working with files in this way.

  • 2
    I can tell you from experience that reading from and writing to large files with PHP (1 MB isn't really that large, but still) can be tricky (and slow). You can always lock the file, but it'd probably be easier and safer just to use a database. – NullUserException Oct 26 '12 at 17:15
  • I know it would be better to use a DB. Please read the question (last paragraph number 4) – hozza Oct 26 '12 at 18:32
  • 2
    I did read the question; I'm saying it's not a great idea and there are better alternatives. – NullUserException Oct 26 '12 at 18:35
  • 2
    file_put_contents() is just a wrapper for the fopen()/fwrite() dance, LOCKEX does the same as if you'd call flock($handle, LOCKEX). – yannis Oct 26 '12 at 19:12
  • 2
    @hozza That's why I posted a comment, not an answer. – NullUserException Oct 26 '12 at 19:30
4

1) No 3) No

There are several issues with the original suggested approach:

Firstly, some UNIX-like systems such as Linux may not have locking support implemented. The OS does not lock files by default. I have seen the syscalls being NOP (no-operation), but that's a few years back, so you need to verify whether a lock set by your instance of the application is respected by another instance. (i.e. 2 concurrent visitors). If the locking is still unimplemented [very likely it is], the OS lets you overwrite that file.

Reading large files line-by-line is not feasible for performance reasons. I suggest using file_get_contents() to load the whole file into memory and then explode() it to get the lines. Alternatively, use fread() to read the file in blocks. The aim is to minimize the number of read calls.

In regards to file locking:

LOCK_EX means an exclusive lock (typically for writing). Only one process may hold an exclusive lock for a given file at a given time. LOCK_SH is a shared lock (typically for reading), More than one process may hold a shared lock for a given file at a given time. LOCK_UN unlocks the file. Unlocking is done automatically in case you use file_get_contents() http://en.wikipedia.org/wiki/File_locking#In_Unix-like_systems

Elegant solution

PHP supports data stream filters which are intended for processing data in files or from other inputs. You may want to create one such a filter properly using the standard API. http://php.net/manual/en/function.stream-filter-register.php http://php.net/manual/en/filters.php

Alternative solution (in 3 steps):

  1. Create a queue. Instead of processing one filename, use the database or other mechanism to store unique filenames somewhere in pending/ and processed in /processed. This way nothing gets overwritten. The database will be also useful for storing additional information, such as metadata, reliable timestamps, processing results, and other.

  2. For files up to a few MB, read the whole file into memory and then process it (file_get_contents() + explode() + foreach())

  3. For larger files read the file in blocks (i.e. 1024 Bytes) and process + write in real-time each block as reading (careful about the last line which doesn't end with \n. It needs to be processed in the next batch)

  • 1
    "I have seen the syscalls being NOP (no-operation) ..." which kernel ? – Massimo Apr 12 '17 at 23:08
  • 1
    "Reading large files line-by-line is not feasible for performance reasons. I suggest using file_get_contents() to load the whole file into memory ..." This is a non-sense. I can say : for performance reasons don't read big files into memory ... What to do depends on many other factors. – Massimo Apr 12 '17 at 23:09
4

I know this is ages old, but in case someone runs into this. IMHO the way to go about it is like this:

1) Open the original file (e.g. original.txt) using file_get_contents('original.txt').

2) Make your changes/edits.

3) Use file_put_contents('original.txt.tmp') and write it to a temp file original.txt.tmp.

4) Then move the tmp file to the original file, replacing the original file. For this you use rename('original.txt.tmp', 'original.txt').

Advantages: While the file is being processed and written to the file is not locked and others can still read the old content. At least on Linux/Unix boxes rename is an atomic operation. Any interruptions during the file writing don't touch the original file. Only once the file has been fully written to disk is it moved. More interesting read on this in the comments to http://php.net/manual/en/function.rename.php

  • Exactly what I wanted to propose as well. But I also would acquire a shared lock while reading to prevent data clobber. – d3L Apr 16 '17 at 6:02
1

In PHP documentation for file_put_contents() you can find in example#2 the usage for LOCK_EX, puting simply:

file_put_contents('somefile.txt', 'some text', LOCK_EX);

The LOCK_EX is a constant with an integer value than can be used on some functions in a bitwise.

There are also a specific function in order to control locking for files: flock() manner.

  • While this is interesting and could be useful in some situations, when reading, modifying and rewriting a file, the lock should be acquired before you read it and maintained until it has been entirely rewritten (otherwise another process may read an old copy and change it back after your process finished). I don't believe this can be achieved with file_get/put_contents. – Jules Apr 16 '17 at 18:06
0

An issue you didn't mention that you also need to be careful of is race conditions where two instances of your script are running at the nearly the same time, for example this order of occurrences:

  1. Script instance 1: Reads file
  2. Script instance 2: Reads file
  3. Script instance 1: Writes changes to file
  4. Script instance 2: Overwrites first script instance's changes to file with its own changes (since at this point its read has become stale).

So when updating a large file, you need to LOCK_EX that file before you read it and not release the lock until the writes have been made. In this example I believe that will cause the second script instance to hang for a little bit while it waits its turn to access the file, but this is better than lost data.

protected by gnat Apr 16 '17 at 7:03

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.