How do I write tests against an eventually consistent service?

Question

I'm building a service on top of Google App Engine Datastore, which is an eventually consistent data store. For my application, this is fine.

However, I'm developing tests that do things like PUT object and then GET object and checking properties on the returned object. Unfortunately, because the datastore is eventually consistent, these simple tests aren't reproducible.

How do you test an eventually consistent service?

Answer 1

Consider non-functional requirements when designing your functional tests -- if your service has a non-functional requirement of "Consistent within x (seconds/minutes/etc)", simply run the PUT requests, wait x, then run the GET requests.

At that point, if the data has not 'arrived' yet, you can consider the PUT request to be non-conformant with your requirements.

Answer 2

OK, so. "What Are You Testing" is the key question.

• I am testing my internal logic of what happens assuming the google stuff works

In this case you should mock the google services and always return a response.

• I am testing my logic can cope with with the transient errors I know google will produce

In this case you should mock the google services and always return the transient error before the correct response

• I am testing that my product will actually work with the real google service

You should inject the real google services and run the test. But! The code that you are testing should have the Transient Error handling (retry) built into it. SO you should get a consistent response. (unless google is very badly behaved)

Answer 3

You really want your tests to be fast and consistent. If you start creating tests that may occasionally fail due to eventual consistency, you'll ignore the test when it fails, and then what use is it?

Create a fake service which handles the PUT and GET requests, but has an additional operation to make it consistent.

class TestDataStore:

  def __init__(self):
    self.current = None
    self.pending = self.current

  def do_put(self, obj):
    self.pending = obj

  def make_consistent(self):
    self.current = self.pending

  def do_get(self):
    return self.current

Your test is then:

datastore.do_put(myobj);
datastore.make_consistent();
validate(datastore.do_get(), myobj);

This allows you to test your software's behavior when the GET successfully retrieves the PUT object. It also allows you to test your software's behavior when the GET does not find the object (or the correct object) due to the service not yet being consistent. Just leave out the call to make_consistent().

It is still worth having tests that interact with the real service, but they should run outside your normal development workflow, as they will never be 100% reliable (e.g. if the service is down). These tests should be used to:

1. provide metrics on average and worst case time between a PUT and a subsequent GET becoming consistent; and
2. verify that your fake service behaves similarly to the real service. See https://codewithoutrules.com/2016/07/31/verified-fakes/

Answer 4

Use one of the following:

• After PUT, retry GET N times until success. Fail if no success after N tries.
• Sleep between PUT and GET

Unfortunately, you have to pick magic values (N or sleep duration) for both of these techniques.

Answer 5

As I understand it, the Google Cloud datastore allows both strongly consistent and eventually consistent queries.

The trade-off is that strongly consistent queries are pretty severely rate-limited (something you can live with during testing).

One possibility may be to put your queries to the datastore within a wrapper that can enable strong consistency for testing purposes.

For example, you could have methods called start_debug_strong_consistency() and end_debug_strong_consistency().

The start method would create a key that can be used as an ancestor key for all subsequent queries, and the end method would delete the key.

The only change to the actual queries you are testing would be to call setAncestor(your_debug_key) if that key exists.

Answer 6

One approach, which is nice in theory but might not always be practical, is to make all write operations in the system under test idempotent. That means, assuming your test code tests things in a fixed sequential order, you can retry all reads and all writes individually until you get the result you are expecting, retrying up until some timeout that you define in the test code gets exceeded. That is, do thing A1, retrying if necessary until the result is B1, then do thing A2, retrying if necessary until the result is B2, and so on.

Then you don't need to bother to check for preconditions of write operations, because the write operations will be already checking them for you, and you just retry them until they succeed!

Use the same "default" timeouts as much as possible, that can be increased if the whole system gets slower, and override the defaults individually when retrying particularly slow operations.

Answer 7

A service such as Google App Engine Datastore is based upon data replication across several globally spread points of presence (POP). Any integration test for an eventually consistent service is really a test of that service's rate of replication across its set of POPs. The rate at which content is spread to every POP in a given service is not going to be the same to every POP within the service depending upon a number of factors, such as the method of replication and various Internet transport issues — these are two examples which account for a majority of reports in any eventually consistent datastore service (at least that was my experience while I was working for a major CDN).

In order to effectively test an object's replication across a given platform you'd need to set the test to request the same recently placed object from specifically each of the service's POPs. I'm suggesting testing the POPs list one-to-five times or until all POPs in your POPs list reports having the object. Here's a set of intervals at which to perform the test which you are free to adjust: 1, 5, 60 minutes, 12 hours, 25 hours after placing it on the datastore. The key is logging the results at each interval for later review and analysis in order to get a feel for a given service's ability to globally replicate objects. Often datastore services only pull a local copy to a POP once it has been requested locally [the routing is done via BGP protocol which is why your test has to request the object from each specific POP for it to be globally valid for a given platform]. In the case of Google's Datastore you'd be looking at setting up your test to query a given object from "over 70 points of presence across 33 countries"; you'd likely have to get the POP specific address url list from Google Support [ ref: https://cloud.google.com/about/locations/ ] or if Google is using Fastly for replication, Fastly Support [ https://www.fastly.com/resources ].

A couple of advantages of this method: 1) You'll get a feel for a given service's replication platform, know its strenths and weakpoints as a whole on a global scale [as it was during the integration test]. 2) For whatever object you test you'll have a tool available to warm content [make that first request which creates the copy at a given local POP] — thus providing you a way to ensure content is spread globally before your clients request it from anywhere on earth.

Answer 8

I have experience with Google App Engine Datastore. Running locally, surprisingly, it often is more "eventually" than "consistent". The simplest example: create a new entity, and then retrieve it. Often times over the last 5 years, I've seen the locally-running SDK not find the new entity immediately, but find it after about half a second.

However, running against the real Google servers, I have not seen that behavior. They try to make your Datastore client always run against the same server on their side, so usually any changes are immediately reflected in queries.

My advice for integration tests is to run them against the real servers, and then you'll probably not need to put any fake polling or delays to get your results.