TDD when testable requirements are hard to come by

Question

I'm in a funny situation. We have a development model that works for an in-house software suite which is focused on responsiveness to the customer. To give you a sense of the environment, when we were using SCRUM, we would often intentionally commit to only 50% of our maximum workload, with a full expectation that the remaining 50% would be filled with quick response work.

The development community at my workplace as a whole is moving more towards TDD, so I have an interest in figuring out how it might be adapted to fit my workflow. That will make it easier to poach other employees, I mean, facilitate the smooth re balancing of resources.

However, I find myself regularly faced with situations that do not have testable requirements until the majority of the work is completed. Because of the rapid response nature of the job, finding the solution that can be done is more important than holding to an arbitrary API picked at the start of the task. It's of no use to my customer if I can build their dream API in a month, when they need to have a working product in a week. The API only really gets finalized after we understand how the code is actually going to do the task.

In situations where testable requirements are hard to come by, has anyone had success leveraging TDD in a meaningful way, even if it's not the "traditional" way?

Answer 1

If you think you can't test without requirements then you don't understand the purpose of tests.

I'm in a funny situation.

No you aren't. This is typical of many shops that use TDD.

To give you a sense of the environment, when we were using SCRUM, we would often intentionally commit to only 50% of our maximum workload, with a full expectation that the remaining 50% would be filled with quick response work.

I've been in shops where that was generous. It's a different world when you have operational systems with up-time requirements. Scrum and TDD still work fine if you do them right.

However, I find myself regularly faced with situations that do not have testable requirements until the majority of the work is completed. Because of the rapid response nature of the job, finding the solution that can be done is more important than holding to an arbitrary API picked at the start of the task. It's of no use to my customer if I can build their dream API in a month, when they need to have a working product in a week. The API only really gets finalized after we understand how the code is actually going to do the task.

Who convinced you keeping a death grip on an API regardless of your needs was traditional?

In situations where testable requirements are hard to come by, has anyone had success leveraging TDD in a meaningful way, even if it's not the "traditional" way?

I have had success with TDD when the only requirement I had was, "I wonder what this thing does".

Tests are not requirements. You might have a requirement. You might write a test that proves that requirement is met. You might even mention that requirement number in the comments of the test (or, God help us, in its name). Fine. But if that's all you're doing with tests you don't know what you're doing. Tests are for more than that. Much more.

I write tons of tests. More than I ever even check in. Why? Because tests help me read code. I write up test after test and watch my understanding of what I can do grow and grow. I change code and I change tests, rapidly. Because I'm learning. The tests lock down what I comprehend now. They remind me what I was doing when I come back from the bathroom.

So yeah, sometimes I write tests without ever looking at or thinking about a requirements document.

I dump a bunch of tests as my understanding grows. I find better ways to express what I'm trying to say. Just like I come back to this answer and rewrite it and move things around I do the same to my code and my tests trying to find the best way to express an idea. Not because the CPU needs me to. Not because requirements demand it. Not because TDD told me to. Hell I was doing this before TDD was a thing. I do it for the humans. I'm looking for the best way to present an idea so that the clueless newbie that comes after me can easily find the one line they need to change because the world changed after I left. I do that by using code and tests to tell the story of the idea well.

The purpose of tests is to help people read code.

They show what it does, what it needs, how much code you have to read to understand what is happening, but they don't tell you what you need. Your needs will change when the tests haven't been touched. It's up to you to decide if the code does what you need. Not the tests.

When you do figure out some requirements you can write other tests for them. You can show them off to your product owner. You can even put them in a special place so you can tell them apart from the ones you just write to help yourself read the code. That's BDD.

So don't complain that you can't write tests because your requirements are up in the air. That describes over half of everything I've done using TDD. Any code you write can be put under test. That test is "I think the code will do this, let's see if it does". That tells future coders (including me) what I was thinking. But deciding if that is needed isn't the tests job. That's yours. If you change your mind it's time to make changes. Change the test and then change the code. That's TDD.

Answer 2

I find myself regularly faced with situations that do not have testable requirements until the majority of the work is completed.

I would argue that it is impossible to finish any portion of a piece of work if you do not have testable requirements for it, because there is a literally zero percent chance that it will be what the customer wants. That said, the requirements may not be unit testable, but that is a good thing - in any high level piece of work you want the requirements to be high level so that you have flexibility during implementation. If the requirements were unit testable you would be right back to a waterfall workflow, where everything is specced up front and you end up with something hard to develop and terrible to use (because up front design does not account for any surprises or learning more about the problem as you work on it).

Now, once you have your high level requirement it's your job to figure out what the next step is. If any part of the work is risky you'll want to do that first. Figure out a high level test (acceptance or even journey level) that, when passing, proves to you that the risk is resolved (for example, your API returns an encrypted version of the input in less than 5 ms). To implement that testable goal you'll discover integration and then unit level testable pieces of work, write tests for them, and continue in standard TDD fashion. Once all of the unit and integration level tests pass your higher level test should also pass, and you have a requirement done and well tested.

Answer 3

If you need to do some experiments and prototyping to find out the correct requirements and a working implementation, a "test first" approach to automated testing may not be working well.

However, writing unit tests and other kind of automated tests afterwards will still work. Sure, this is not "TDD by the book", but the design impact on your code will typically stay almost the same as if you had written the tests beforehand.

Answer 4

If you write tests because you have requirements it seems wrong o me. During my application developing process I also write a lot of tests as they help me to test my code in a small and defined environment. Without this approach I would have to run the entire application which is very time consuming.

Answer 5

There are decent pieces of information in these answers but I'm not seeing much really actionable information. I'm not a testing expert but the way the way I understand TDD in a nutshell is that you follow this process:

• Write a test
• Make sure it fails for the right reason
• Write code to make the test pass

'Testing' is a fairly broad topic but I think you can apply the above easily to unit tests and also acceptance tests.

In the case of unit tests, requirements don't really matter IMO because unit tests don't exist to prove the code meets one or more requirements. Unit tests are written to prove your code works the way you meant it to work. That's it. If I have an add() method, I should test that it adds correctly. It might be the case that I really needed a multiply method. That's not something you worry about in unit testing. Therefore, a lack or requirements should not be a barrier to unit testing. It's part of the coding process.

As far as acceptance testing goes, one approach that is pretty 'test-driven' is writing the requirements as executable tests. The delivery of requirements is then "make these tests work." This is very cool if you can get the process going but the time to write these tests is significant and can become an development effort in it's own right. I'm not saying it's not worth it. If you can get it working, the benefits are tremendous. It might be tough though if you are playing 'find a rock':

[you] here's a rock

[them] not that rock.

You will be spending a lot of time writing and rewriting the test scripts. You can still do it, but it could be costly. Essentially, you make a guess at what you are supposed to deliver and put that into the tests. You write the code to make that test work and when the 'deciders' tell you they wan't something else, you modify the tests, rinse and repeat.

If you aren't doing TDD unit testing, it's a lot easier to get things going that way but I have a sense you are talking about something more like acceptance testing here.

Answer 6

Can you change the culture? It sounds like you will need to in order to iteratively develop the UAT and refine the requirements while developing the code.

It depends on the organization and how the organization determines the definition of done. Point out that if they want to increase their software quality, there needs to be changes in how developers and customers interact.

In the most functioning companies that I have worked for, refining the requirements was up to the the senior technical staff and the business analysts.

Answer 7

It's hard to determine exactly what you mean, but I've worked in a place who insisted that they couldn't deliver testable requirements, I'll assume you mean similar.

In this environment, my problem was more "I don't really know exactly what they're asking me to build", which was why it was very hard to write tests for it. Requirements were vague, and everyone was just sortof muddling towards an ever-changing target each day.

They also called this "responsive", but I now know it was just a manifestation of amateurism and confusion.

In practically any object-orientated programming team, you should know what you're building in advance. You should always be starting with a requirement against which you can at least write a unit test, even if you can't write the implementation.

If someone asks you to do something, you can write a test. It's that simple. If you cannot imagine a way to write the unit test you're not really being given a requirement. If all you've known is one of those "muddling through" employers (which isn't uncommon) this might be hard to grasp, but requirements in a well-built, object-orientated system should almost always equate to a particular behaviour exhibit by a particular object given a particular set of inputs (the S of the SOLID principle), and it is against that object that you can write your test.

If doing this seems completely impossible, that's a process smell--a huge one.

Answer 8

There are always requirements, otherwise you would have no reason to write code.

Some times those requirements are given to you, sometimes as a programmer you have to make up the requirements yourself using your understanding of the aims of the organisation you work for. In practice it's generally a mixture of the two.

Wherever the requirements come from, if they are specific enough to enable you to write some code, then you should also be able to use them to write tests if you want to.

In some ways tests are particularly important if you have made up some of the requirements and are likely to need to change them, since the tests may then be the only documentation of those requirements, and are a place to show any changes to requirements that might happen after you get feedback on a version of the code.