We are trying to improve our continuous integration pipeline and moreover improve how quickly we can get code through from inception to production.

We have tests at different levels that verify what our software is supposed to do functionally e.g. if there were 20 primary use cases, we have those 20 primary use cases covered with tests.

The question I have is about the configuration data that sits in the database. So for example we could have 200 customer configuration profiles configured in the database. How do people "verify" the data configured in the database is correct?

Do we have to carry the burden that for each customer, we must write tests that verify that the customer configuration is correct? Or is configuration data (in particular configuration data that exists en masse) outside the scope of continuous integration tests?

  • 1
    You are not validating data integrity through CI. You should be capabale to execute the tests no matter what the current status of the data (in db) is. Your are not validating 200 configurations, you are testing "a user with its Configuration" no matter which one is. And you should be able to recreate such scenario anytime your tests are executed. – Laiv Nov 8 '16 at 17:53
  • Thanks for the info. I agree that validating data integrity shouldnt be part of CI. However in that case how is the configuration validated?, Is it manual?, Have you any reference points like documentation etc? – bstack Nov 9 '16 at 8:31
  • It's unclear what exactly you are asking for. If you need verification of data integrity then you need to write a test/specification for the data/configuration format. If you are using a SQL database then the schema already provides validation. If you're not using a SQL database then data validation now becomes code that you will need to implement. – davidk01 Nov 23 '16 at 18:57

If you mean "how do I test that customer x has the correct configuration"..

This is done by implementing some form of change control on the configuration. Either have it in source control or some system which keeps a history and audit trail of changes.

Knowing what the 'correct' configuration is, is 90% of the task. Once you have that you can simply compare with the deployed version.

If you mean "how do I know feature x works with configuration y" then the answer is "moarr tests!!!" (which is why feature toggles are bad)

edit - when is it enough?

You can write tests until the end of days but at some point its not worth it. Record and review your bugs and mistakes. Are there too many?

  • We perform change control of our configuration data via git and we have a patch management process. Is this enough?, Do we also require that we have automated tests that executes the software for each of the e.g. 200 customer configurations. – bstack Nov 9 '16 at 8:34
  • this is why I hate feature toggles and complicated configurations. every switch you have, doubles the number of tests you should run – Ewan Nov 9 '16 at 8:57
  • 1
    Feature toggles aren't bad. Long lived feature toggles might be though. Kind of like long lived branches... – RubberDuck Nov 9 '16 at 10:55
  • at least branches end before they go live. feature toggles are BAD! (but that is one of many arguments best carried out in a pub) – Ewan Nov 9 '16 at 12:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.