How to reset state of exernal services when running Acceptance tests

Question

What are the generic patterns and best practices for resetting state of a database, storage, external service, etc, in Acceptance Tests?

I'm struggling with spaghetti, living in acceptance tests. It connects to the remote DB, executes SQL, Rebuilds buckets through AWS S3 API, reaches into a payment provider API to delete payments and so on. It does this between every scenario, so the tests are isolated.

This is specific for acceptance tests, that test the integration of a service as a whole and the data flow between some services (integration of multiple services). There is hexagonal service, which has an HTTP API, HTML UI, CLI UI on one side and some services like a database, file-storage, payment service, SMTP, push notification etc on the other side. Many of these services need to be reset, as they affect the state of the app and thus how acceptance tests can run.

My current strategy makes the acceptance tests tightly coupled to implementation details which, as far as I know, the tests should best be unaware of.

But it also introduces practical difficulties, for example, I now need to open the database server to allow connections (which can delete data, schema's etc) from "random" test runners "anywhere" on the web (I'm using Github Actions, but the same would apply to AWS pipelines or CircleCI or most hosted test-runners). So this also has security implications, makes it hard to migrate and so on.

Strategies I'm considering are:

• Ensure the acceptance tests use unique (random) names and attrs . No need to reset.
• Move the reset-logic into the app into a public "reset interface".
• Keep reset-logic in the accptance tests but refactor them to be less spaghetti. Deal with the practical problems with e.g. VPN or proxies.

Yet I'm certain more people have this problem. And certain that is has been solved multiple times already. So I'm mostly looking for any hints, talks, articles, or books that explain solutions and their tradeoffs.

Answer 1

TL;DR
I think you've analyzed your options well; the main takeaway from this answer is that it's perfectly okay to have some logic in your test suite related to setup/cleanup that does not count towards "test logic" in and of itself, and therefore can circumvent the general "tests should not know implementation" good practice advice.

My current strategy makes the acceptance tests tightly coupled to implementation details which, as far as I know, the tests should best be unaware of.

Note that any logic you have to write in your test suite to prepare or clean up the data is not part of "the tests" in the above statement.

For example, if you have a database cleanup method that empties the database, it's perfectly fine for that to bypass your domain logic entirely. Its purpose is not to test, but rather to provide a tangential meta-purpose that is unrelated to the tests themselves.

Ensure the acceptance tests use unique (random) names and attrs . No need to reset.

It's preferable to not have to do this, simply as a matter to skip the needed effort and avoid possible collisions from occurring if you make mistakes in your collision avoidance.

However, if the cost of setup and cleanup is prohibitively expensive for your test suite and it is reasonably easy to implement a collision avoidance approach; this can be a valid approach.

In short: try to avoid unless there's very good reason to do so.

Move the reset-logic into the app into a public "reset interface".

It doesn't have to be part of the application. If it is not used in the real application, it doesn't have to be added to the application code.

I wouldn't say it's dead wrong to add it if you so choose, but I do think it's relevant to point out that it doesn't have to be in the application logic either.

This is mostly a practical consideration. Does it significantly pollute the application logic by adding it? Does adding it help keep things consistent and prevent you from forgetting to update the test suite if your data model happens to change? Is the effort of adding it into the application code worth the benefits you stand to gain? ...

Keep reset-logic in the accptance tests but refactor them to be less spaghetti. Deal with the practical problems with e.g. VPN or proxies.

Note that it's perfectly fine for your test setup/cleanup logic to be abstracted into a (small) set of classes for that specific purpose. While you want to avoid any excessive logic here (because that would warrant tests in and of themselves), there's nothing wrong with having setup/cleanup logic that is more than trivial.

For example, for a project that used a really data-heavy setup (payroll platform) we used to have JSON files that defined specific scenarios. Each test would denote which scenario it wanted to load (scenarioLoader.Load("my_scenario_bla.json), and there would then be a relatively complex bit of logic that would fetch the JSON data and perform a rather large data insertion into the integration database.

Note that in this particular case, the loader bypassed the application domain itself and opted for direct data insertion into the tables, because it was a whole lot easier to do than going through the domain with its intricate pathways and rigorous access control.

The company, being a stickler for rigorous testing, opted to integration test their scenario-loading logic as well. If your setup/cleanup logic is sufficiently non-trivial and you prefer to specifically cover it with tests, you can.

---

To summarize, I think you've analyzed your options well; the main takeaway from this answer is that it's perfectly okay to have some logic in your test suite related to setup/cleanup that does not count towards "test logic" in and of itself, and therefore can circumvent the general "tests should not know implementation" good practice advice.

Answer 2

A bit of a frame challenge: are your tests actually coupled to the implementation?

You are testing a piece of software in an environment of several external services. Neither of them is part of the implementation in question - and so aren't their contents.

Perhaps it can be argued that e.g. the database is agnostic of the data scheme, but that doesn't really matter. Clearing database in that case would be most likely data scheme agnostic, too. OTOH, if you need to initialize the database with some fixture, that fixture is really a part of the test, not the cleanup.

If you consider the schema an implementation detail, you should* populate the database using the API. However, if anything else has access to the database and can read the data, the schema is part of the public API, really. And in that case, populating the database with a fixture is depending on API, not on implementation details.

The same or similar logic applies to all other dependencies: is their setup an implementation detail, or rather just modelling a life-like environment for the service?

I guess all that I'm really trying to say, is that there is a distinction between implementation details of the service under test and system as a whole. When integration testing the service, you need to avoid reliance only on the former - in a sense, you are testing its fit to the latter, so you want to use some implementation detail of the system.

* if you want to avoid dependency on implementation detail, that is.

As for solution: depending on the scale and how much prod-like the environment is required to be, it may be feasible to simply tear down all services after every test, and then sping them up fresh for the next. Bonus points for using containers or VMs for running the services, so that they really start fresh every time. That will add runtime, but should give strong guarantees about the cleanup, way better than some bespoke cleanup operation. Using some pre-baked VM or container images with things set up may help. Replacing some services with lighter alternatives may help too, but it has a downside of not actually doing the integration, so you need a lot of trust that the behavior doesn't differ.

While it may be costly to implement, this may be beneficial to the whole project too, as the solution for spinning up all the services may be the same one as used in production, just with slightly different configuration.