How do we design a test approach for inexperienced testers?

Question

We're in the process of shifting our datasource from Mongo to MS SQL Server. Our challenge is that we have two testers who are green. As a team, we're trying to work through the most sensible test approach.

We have two datasources (one MS SQL, one Oracle) that are integrated via flat file feed to a MongoDB, which is used to present product data on a website. Our endpoint is to convert the MongoDB to MS SQL.

We have two pieces to test.

1. Datasource comparison. The data feed is loaded into both sources. Our DBA installed Simba MongoDB ODBC with SQL Connector driver to allow us to make side-by-side comparisons using SQL. It seems to me the biggest risk here is that the driver does its own set of interpretations about the underlying structure of documents in the Mongo source in order to present the data as tables, and on top of that, our queries also need to reshape the data to make sure we can compare SQL and Mongo sources.

2. Display logic comparison. The data objects that drive the product pages are theoretically source-agnostic, but there is display logic in the front end that could be impacted if the SQL data model and the Mongo data model aren't aligned exactly -- and they aren't. We've installed HtmlAgilityPack in order to compare elements on the product pages using C#. I'm not sure what the risks are here.

Given the additional considerations that:

• We are moving the Oracle DB to MS SQL this year and will use this test approach for that project.
• We are in desperate need of automated testing for our product pages.
• We have a plan to normalize the SQL DB source in phases after the cutover, and then to eliminate the data feeds, all of which will require robust datasource/display logic comparisons.
• Our testers have exactly zero experience with testing data and web interfaces, and also have no development background that will enable them to write code for automated testing.

My concerns are that we may be considering the wrong approach because the testers do not have sufficient experience to tell us what they can do. The dev group knows how to test this way, so that's the way we're designing it. I don't know if this approach will work and be maintainable through next year's releases without significant investment of non-tester time and resource, and if it does require non-tester time and resource, if that continued investment worth it if it means the testers end up doing work that requires no thinking.

Is there another way to do this that could be used by our testers, to enable them to have more ownership over the test process, support their skill development, and also provide automation for future releases?

Answer 1

Our testers have exactly zero experience with testing data and web interfaces,

No real problem, they will get used to it ...

and also have no development background that will enable them to write code for automated testing

That is your main problem, but let me explain.

Manual testing can be very useful for UI applications when your testers know the business and the requirements well, and you are implementing new "user stories" or requirements one by one which must be systematically validated and checked for their correctness.

However, for what you have in mind you need mainly regression tests - which means after a change "under the hood" at the database level, the front end parts (ideally all parts) of the application must be executed, and one has to validate that the behaviour of the old application is still the same. If the behaviour is "right or wrong" in the business sense will be of minor concern during this phase. "Right" typically means "the old behaviour" by definition.

Of course, this can be done manually: let your testers

• write a test plan how to systematically run the application, so every important db table is accessed, and every important query will be executed (your devs should support the testers for this step)

• let them execute the test plan step-by-step with the old and the migrated application side-by-side, and let them check if the two application versions behave exactly the same

Executing this test plan a few dozen times, whenever a new milestone is reached during your migration, will soon become cumbersome. Furthermore, your devs might want to have these tests executed even more frequently, for example once a day. In this situation, I guess your testers will quickly find out by themselves that it will make much more sense to automate this repeating, boring process. And that is the point where someone should introduce a tool like Selenium or some commercial UI robot tool to automate the whole process: you use the old application to "record" some expected behaviour, and then run a replay-and-compare with the new application. However, one will definitely need some specific experience with the test tools to get this working - just "record and replay" (to my experience) won't work "out of the box". You testers will need some coding experience, some experience how to manage test data and how to create data driven tests.

So train your testers how to utilize these tools. If you do not have any confidence that they can learn how to automate their tests, then the only recommendation I can give you to get a different kind of testers for this work.