1

General overview

We recently had lots of problems with automated tests in our team. Part of it was that the people designated to writing them had little experience. After this failed, we incorporated a different approach where the product owner and the developers are much more engaged in the overall process of automated testing, so the quality improved.

There is still one more problem that we are facing without a good solution and that is the test data and it's maintenance.

We have integration tests written using RestSharp and xUnit (the back end is ASP.Net Core) and GUI tests written with the use of Selenium and xUnit (the front end being in Angular).

Both of these rely on MS SQL as the data store.

Now best practices state that the tests should be as independent as possible from each other. This is quite easy to accomplish in unit tests where we can arrange test data because the tested part is very small. In GUI/Integration tests however, matters are a bit different.

Problem Description

Since GUI/integration tests often perform complicated actions (which test many more elements than unit tests), it is very difficult to create independent mock data in a database for each individual test.

Scenario

Let's assume that we want to write automated tests for some hierarchical organization structure which is added through CRUD screens. We have sites with child areas which then have child subareas and so on.

The first test would be to test adding and editing Sites. This is quite simple. The second test for Areas requires assigning a preexisting Site in the system so we can create hierarchy. Third would require Site and Area and so on.

So basically the more complex the test scenario, the more preexisting specific data it may require and failures at the beginning of the hierarchy-add will fail other tests which may work fine on their own.

Possible solutions

  1. Tests rely on each other for creation of data - This is in my opinion a very bad approach because failing the first test will fail the rest. Plus it doesn't scale very well for more complex scenarios.

  2. Each test loads it's data individually - this allows the tests to be independent but because of this we will have lots of code just to load the data for the test in case of changes or problems it will also cause a lot of rework so it will also make the tests a bit fragile.

  3. We could use containerization with Docker. Create database image and load the database with data on each run. This will allow us to make sure that more tests can be run in parallel but again solution 1 and 2 would also need to be implemented to push the data into the container.

  4. We could divide the tests into different categories. Simple/Complex ones. Simple one we would assume can be written in as much isolation as possible. Complex ones could have shared steps (Hierarchy would be such an example and we could accept that there will be some chained test steps there).

Any guidance here would be greatly appreciated. Creating successful test automation is a lot of work and the best thing is to start in the best direction possible.

  • 1
    IMO there isn't a best answer to this. The real solution would be to maximize the amount of testing that can be done in isolation. Once you minimize the amount of complex, long-running end-to-end tests, this becomes a less of an issue. – Euphoric Feb 20 at 9:12
  • I know there isn't an ideal solution but there might be the best solution. Plus any experiences with the mentioned would be great help. – Mateusz Migała Feb 20 at 9:19
  • My suggestion: use automated test to test a real version of your application (deployed on a separate QA environment maybe). Create one "automated test case" for each "manual test case" that is executed by QA team. Like this you don't need to mock anything. On the automated test, you need to use selenium to simulate the same navigation that the tester does during QA validation, for a specific test case / per environment. – Emerson Cardoso Feb 20 at 10:20
0

Basically in the frames of MVC model, UI and controller layer tested separately from data model. You can use any DI frameworks, or simply Mocks and Spies in order to emulate data sources and, by the way to check reaction of you handlers. If you just test UI,or even Controller, forget about databases (SQL Server,MySQL etc). Use injection of a DAO, which emulates data and assert calls of its methods and data visibility in UI. Otherwise what you a testing: UI, or a data layer?

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.