4

TL;DR: My goal is to create some framework with which I could discover/test all possible internal states of my application so that I would approach with confidence for example engineering software for medical devices, airplanes etc...

Let me first explain some context...

I was watching interesting video on example mapping technique used in agile practices. The video is about how to explore the domain space with examples which are creating a map for the problem domain space. In more concrete words they tried to get to model a solution for user story: "booking a train ticket". After user story was presented, they layout rules or constraints for the solution space. After that they layout some basic examples in which they created interesting cases (examples) in which entities of the domain space can get into... From brainstorming session, they discover problem space that was not known before and created questions like "what should system do if this particular state can occur".

Every programming solution that we programmers create is essentially to create a behavior for a list of use cases. Many times we don't cover all the states in which our solution can get into or we miss unknown use cases that are creating illegal states and this is why further interactions are creating errors (unwanted behavior) of our system.

So I want to incorporate discrete event simulation testing in my standard toolbox that would analyze my system under scope with a mission to explore the problem space and to discover that unwanted behavior. Many times happened before when I didn't have full knowledge about the domain space I was learning about that domain space with producing those errors, so using some discrete event testing simulation will be a great benefit to my software engineering practice along with DDD and TDD.

I'm a big fan of TDD but it often happens that you don't cover all the cases because of limited domain knowledge...

So I'm thinking to incorporate some form of white box testing framework in which I would tell what are internal and external input/output constraints for various input arguments, databases calls, external services calls etc...) and the testing framework should discover unexpected behavior (bugs etc...) instead of me creating all those e2e tests that could be covered by the testing framework. And this could be a great addition alongside TDD...

My question is how NASA, for example, test their rocket system software? They must incorporate some form of discrete event simulations on their software so that they are confident that they have explored all the states in which system can enter?

  • 3
    No, they do not do anything like that. As soon as your program stores about 10 bytes of data you have more combinations of state than atoms in the universe. Brute forcing them is infeasible. – Telastyn Dec 1 '18 at 20:12
  • 1
    You might get a better answer over at Software Quality and Analysis but I have an answer for this over at (surprisingly?) Workplace.SE - see halfway down at "JPL". – Rob Dec 1 '18 at 23:10
  • 1
    There is an article about how NASA writes mission-critical software that goes into some depth about how their process works here: fastcompany.com/28121/they-write-right-stuff – Robert Harvey Dec 2 '18 at 18:18
13

I've worked in places that produce life critical software. We do not "test all possible internal states". Instead we:

  • Review all code before accepting it into the code base (see Peer Review & Security Review)
  • Test each side of every boundary in our state space (see Boundary Testing)
  • Test all paths through the code (see Code Coverage)
  • Test the readability, test-ability, and correctness of the code by having peer reviewers supplement the authors unit tests with their own (see Peer Testing)
  • Gather testable requirements that QA can ensure have been met.

That might sound like waterfall but in my experience as long as you can do all that on one tiny feature within two weeks then people are content to call it agile.

Personally I like one week sprints but every team is different.

  • +1 Good answer. One nitpick thought: "that might sound like waterfall"? Why would anyone think testing techniques and QA have anything to do with waterfall? These are orthogonal concepts! – Andres F. Dec 2 '18 at 5:45
  • 1
    @AndresF. just something I continually find myself needing to pointing out. Some think this level of testing is unique to waterfall when what's really unique to waterfall is a desire to force requirements to change less than 3% over the life of the project. – candied_orange Dec 2 '18 at 6:00
  • Thanks for recommendations, this is much appreciated. "Test each side of every boundary in our state space (see Boundary Testing)" Yes this is something I'm really interested! Can you elaborate on your answer a little bit more on how your team approached this practice, and I would really like to know about the software that you talking about, please make this happen and I will accept your answer. – user157581 Dec 2 '18 at 7:33
  • Oh and did you test internal states (like external services, DB, etc...) of the software with boundary testing too or joust public inputs of the software? – user157581 Dec 2 '18 at 7:39
  • @UrošJarc boundry testing doesn't care what the source of input is. It cares about which values should produce different behaviors. – candied_orange Dec 2 '18 at 8:57
1

@candied_orange 's answer is great. In addition, you need good requirements and test cases first. I recommend taking a look at Cucumber and spec flow and the like for domain communication.

Usually, good and meticulously designed test cases, clear requirement analysis, %100 test coverage in unit and integration tests plus boundary cases and a well documented and cleanly read code should be fine, even for life critical systems.

If you want to go further, (which you do it seems, in your "TL;DR" note) I can recommend mutation testing. Basically, it consists of randomly altering the source code, test code and assertions, making sure that the cases conflicting with the unaltered do not pass. You still need to cover all boundary cases manually of course. This method helps you identify, for instance, irrelevant if conditions. This one is a framework for java: http://pitest.org

  • Yea this video that I posted in the links was from cucumber blog, :) – user157581 Dec 2 '18 at 7:28
  • Oh and thanks for recomendation, this is much appreciated. – user157581 Dec 2 '18 at 7:29

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy