PHP file_put_contents File Locking

Question

The Senario:

You have a file with a string (average sentence worth) on each line. For arguments sake lets say this file is 1Mb in size (thousands of lines).

You have a script that reads the file, changes some of the strings within the document (not just appending but also removing and modifying some lines) and then overwrites all the data with the new data.

The Questions:

1. Does 'the server' PHP, OS or httpd etc. already have systems in place to stop issues like this (reading/writing half way through a write)?

2. If it does, please explain how it works and give examples or links to relevant documentation.

3. If not, are there things I can enable or set-up, such as locking a file until a write is completed and making all other reads and/or writes fail until the previous script has finished writing?

My Assumptions and Other Information:

1. The server in question is running PHP and Apache or Lighttpd.

2. If the script is called by one user and is halfway through writing to the file and another user reads the file at that exact moment. The user who reads it will not get the full document, as it hasn't been written yet. (If this assumption is wrong please correct me)

3. I'm only concerned with PHP writing and reading to a text file, and in particular, the functions "fopen"/"fwrite" and mainly "file_put_contents". I have looked at the "file_put_contents" documentation but have not found the level of detail or a good explanation of what the "LOCK_EX" flag is or does.

4. The scenario is an example of a worst case scenario where I would assume these issues are more likely to occur, due to the large size of the file and the way the data is edited. I want to learn more about these issues and don't want or need answers or comments such as "use mysql" or "why are you doing that" because I'm not doing that, I just want to learn about file read/writing with PHP and don't seem to be looking in the right places/documentation and yes I understand PHP is not the perfect language for working with files in this way.

Answer 1

I know this is ages old, but in case someone runs into this. IMHO the way to go about it is like this:

1) Open the original file (e.g. original.txt) using file_get_contents('original.txt').

2) Make your changes/edits.

3) Use file_put_contents('original.txt.tmp') and write it to a temp file original.txt.tmp.

4) Then move the tmp file to the original file, replacing the original file. For this you use rename('original.txt.tmp', 'original.txt').

Advantages: While the file is being processed and written to the file is not locked and others can still read the old content. At least on Linux/Unix boxes rename is an atomic operation. Any interruptions during the file writing don't touch the original file. Only once the file has been fully written to disk is it moved. More interesting read on this in the comments to http://php.net/manual/en/function.rename.php

Edit to address commments(too for comment):

https://stackoverflow.com/questions/7054844/is-rename-atomic has further references to what you might need to do if you are operating across filesystems.

On the shared lock for the reading I am not sure why that would be needed as in this implementation there is no writing to the file directly. PHP's flock (which is used to get the lock) is a little but unreliable and can be ignored by other processes. Thats why I am suggesting using the rename.

The rename file should ideally be named uniquely to the process doing the renaming so as to make sure not 2 processes do the same thing. But this of course does not prevent editing of the same file by more than one person at the same time. But at least the file will be left intact (last edit wins).

Step 3) & 4) would then become this:

$tempfile = uniqid(microtime(true)); // make sure we have a unique name
file_put_contents($tempFile); // write temp file
rename($tempfile, 'original.txt'); // ideally on the same filesystem

Answer 2

1) No 3) No

There are several issues with the original suggested approach:

Firstly, some UNIX-like systems such as Linux may not have locking support implemented. The OS does not lock files by default. I have seen the syscalls being NOP (no-operation), but that's a few years back, so you need to verify whether a lock set by your instance of the application is respected by another instance. (i.e. 2 concurrent visitors). If the locking is still unimplemented [very likely it is], the OS lets you overwrite that file.

Reading large files line-by-line is not feasible for performance reasons. I suggest using file_get_contents() to load the whole file into memory and then explode() it to get the lines. Alternatively, use fread() to read the file in blocks. The aim is to minimize the number of read calls.

In regards to file locking:

LOCK_EX means an exclusive lock (typically for writing). Only one process may hold an exclusive lock for a given file at a given time. LOCK_SH is a shared lock (typically for reading), More than one process may hold a shared lock for a given file at a given time. LOCK_UN unlocks the file. Unlocking is done automatically in case you use file_get_contents() http://en.wikipedia.org/wiki/File_locking#In_Unix-like_systems

Elegant solution

PHP supports data stream filters which are intended for processing data in files or from other inputs. You may want to create one such a filter properly using the standard API. http://php.net/manual/en/function.stream-filter-register.php http://php.net/manual/en/filters.php

Alternative solution (in 3 steps):

1. Create a queue. Instead of processing one filename, use the database or other mechanism to store unique filenames somewhere in pending/ and processed in /processed. This way nothing gets overwritten. The database will be also useful for storing additional information, such as metadata, reliable timestamps, processing results, and other.

2. For files up to a few MB, read the whole file into memory and then process it (file_get_contents() + explode() + foreach())

3. For larger files read the file in blocks (i.e. 1024 Bytes) and process + write in real-time each block as reading (careful about the last line which doesn't end with \n. It needs to be processed in the next batch)

Answer 3

In PHP documentation for file_put_contents() you can find in example#2 the usage for LOCK_EX, puting simply:

file_put_contents('somefile.txt', 'some text', LOCK_EX);

The LOCK_EX is a constant with an integer value than can be used on some functions in a bitwise.

There are also a specific function in order to control locking for files: flock() manner.

Answer 4

An issue you didn't mention that you also need to be careful of is race conditions where two instances of your script are running at the nearly the same time, for example this order of occurrences:

1. Script instance 1: Reads file
2. Script instance 2: Reads file
3. Script instance 1: Writes changes to file
4. Script instance 2: Overwrites first script instance's changes to file with its own changes (since at this point its read has become stale).

So when updating a large file, you need to LOCK_EX that file before you read it and not release the lock until the writes have been made. In this example I believe that will cause the second script instance to hang for a little bit while it waits its turn to access the file, but this is better than lost data.