What should I do when optimistic locking doesn't work?

Question

I have this following scenario:

1. A user makes a GET request to /projects/1 and receives an ETag.
2. The user makes a PUT request to /projects/1 with the ETag from step #1.
3. The user makes another PUT request to /projects/1 with the ETag from step #1.

Normally, the second PUT request would receive a 412 response, since the ETag is now stale - the first PUT request modified the resource, so the ETag doesn't match anymore.

But what if the two PUT requests are sent at the same time (or exactly one after the other)? The first PUT request does not have time to process and update the resource before PUT #2 arrives, which causes PUT #2 to overwrite PUT #1. The whole point of optimistic locking is for that not to happen...

Answer 1

The ETag mechanism specifies only the communication protocol for optimistic locking. It's the responsibility of the application service to implement the mechanism to detect concurrent updates to enforce the optimistic lock.

In a typical application that uses a database, you'd usually do this by opening a transaction when processing a PUT request. You'd normally read the existing state of the database inside that transaction (to gain a read lock), check your Etag validity, and overwrite the data (in a way that'll cause a write conflict when there's any incompatible concurrent transaction), then commit. If you setup the transaction correctly, then one of the commits should fail because they'll both be trying to update the same data concurrently. You'll then be able to use this transaction failure to either return 412 or retry the request, if it makes sense for the application.

Answer 2

You have to execute the following pair atomically:

• checking of the tag for validity (i.e. is up to date)
• updating the resource (which includes updating its tag)

Others are calling this a transaction — but fundamentally, the atomic execution of these two operations is what prevents one from overwriting the other by accident of timing; without this you have a race condition, as you're noting.

This is still considered optimistic locking, if you look at the big picture: that the resource itself is not locked by the initial read (GET) by any User or any Users who are looking at the data, whether with intent to update or not.

Some atomic behavior is necessary, but this happens within a single request (the PUT) rather than attempting to hold a lock over multiple network interactions; this is optimistic locking: the object is not locked by the GET yet still can be safely updated by PUT.

There are also many ways to achieve atomic execution of these two operations — locking the resource is not the only option; for example, a lightweight thread or object lock may suffice and depends on your application's architecture and execution context.

Answer 3

It's on the application developer to actually check the E-Tag and provide that logic. It's not magic that the web server does for you because it only knows how to calculate E-Tag headers for static content. So let's take your scenario above and break down how the interaction should happen.

GET /projects/1

Server receives the request, determines the E-Tag for this version of the record, returning that with the actual content.

200 - OK
E-Tag: "412"
Content-Type: application/json
{modified: false}

Since the client now has the E-Tag value, it can include that with the PUT request:

PUT /projects/1
If-Match: "412"
Content-Type: application/json
{modified: true}

At this point your application has to do the following:

• Verify that the E-Tag is still correct: "412" == "412" ?
• If so, make the update and calculate a new E-Tag

Send the success response.

204 No Content
E-Tag: "543"

If another request comes and attempts to perform a PUT similar to the request above, the second time your server code evaluates it, you are responsible to provide the error message.

• Verify the E-Tag is still correct: "412" != "543"

On failure, send the failure response.

412 Precondition Failed

This is code you actually have to write. The E-Tag can in fact be any text (within the limits defined in the HTTP spec). It doesn't have to be a number. It can be a hash value as well.

Answer 4

As a complement to the other answers, I will post one of the best quotes in the ZeroMQ documentation that faithfully describes the underlying issue:

To make utterly perfect MT programs (and I mean that literally), we don't need mutexes, locks, or any other form of inter-thread communication except messages sent across ZeroMQ sockets.

By "perfect MT programs", I mean code that's easy to write and understand, that works with the same design approach in any programming language, and on any operating system, and that scales across any number of CPUs with zero wait states and no point of diminishing returns.

If you've spent years learning tricks to make your MT code work at all, let alone rapidly, with locks and semaphores and critical sections, you will be disgusted when you realize it was all for nothing. If there's one lesson we've learned from 30+ years of concurrent programming, it is: just don't share state. It's like two drunkards trying to share a beer. It doesn't matter if they're good buddies. Sooner or later, they're going to get into a fight. And the more drunkards you add to the table, the more they fight each other over the beer. The tragic majority of MT applications look like drunken bar fights.