5

Another name for this question could be: How to TDD the main function?

My situation: I'm writing a program from scratch that runs a sort of simulation, and I want to apply TDD to the whole development process. It seems straightforward to me when I think about the small detailed features, but at some point the program needs to start. For example, this is a rough outline of how the program will run:

  1. Read data from a database.
  2. Loop a bunch of times simulating the desired behavior.
  3. Write the output somewhere.

I'm tempted to write a test like testLoopProducesValidOutput but that's basically just checking to see if the entire program ran correctly, which doesn't seem very unit-like for a unit test.

In the spirit of "don't write code unless you have a failing test", I don't quite understand what kind of test to write to actually get that initial high-level code written that isn't running the entire thing. How would one generally approach this kind of situation?

(Of course one solution is just to not strictly follow TDD for this part, but if there's another solution I'm open to learning before I give up. :) )

3
  • 2
    What are you specifically focusing on testing here? The entry void Main() function (or whatever your language's equivalent entrypoint is)?, or rather the main logic (your bullet points) which happen to be in the main entrypoint? Are you trying to test the entire combined behavior of your application, or only that the main logic (your bullet points) call the rest of its dependencies correctly? Right now, this question is a bit aimless without specifying what it is you want to test. It comes across as wanting to write tests on principle rather than for a known purpose.
    – Flater
    Feb 12, 2023 at 22:42
  • 1
    @Flater I'm testing the main logic. For example, following TDD principles I'd need to write a test before I'm supposed to write that loop logic, but that loop logic basically encapsulates the entire program aside from data I/O. It might even look like for step in iterations: simulate_step(step). While very simple, it's an important part of the logic, but a single unit test that runs this is basically running the entire program. To just test the looping behavior, maybe an answer is just mocking the simulate_step function? Feb 12, 2023 at 23:18
  • 1
    If your testing framework allows it and your program allows it, why shouldn't you check that your processing code produces correct outputs from certain inputs? Feb 13, 2023 at 13:46

7 Answers 7

3

TDD isn't limited to unit tests, it can be applied at any granularity. What you're trying to do is covered by acceptance test-driven development (ATDD) which adds an outer loop to the classic red-green-refactor cycle: you write a failing acceptance test first which should help you come up with a skeleton of your overall program, then you write failing unit tests before finally implementing the actual logic. The acceptance tests ensure that you are building the right thing and your various components are wired together correctly, while the unit tests ensure that your components are implemented correctly.

There is an excellent book on this topic called Growing Object-Oriented Software, Guided by Tests that goes into detail about this process and includes a worked example of a GUI application built from scratch using TDD.

3

I'm tempted to write a test like testLoopProducesValidOutput but that's basically just checking to see if the entire program ran correctly, which doesn't seem very unit-like for a unit test.

You're specifically talking about unit testing, so "let's test the whole lot in one go" is not in the spirit of that. Analyzing this with our unit testing cap on, we can test individual potential points of failure here.

  • Does the main logic call [mocked] database dependency to fetch the data?
  • When [mocked data] is returned from the [mocked] database dependency, does the main logic call the expected [mocked] simulation dependencies the expected amount of times?
  • When a [mocked] simulation dependency returns [mocked data], does the main logic pass [mocked data] into a [mocked] logging dependency?

I've been repetitively explicit about point out that effectively everything except the main logic's behavior itself is mocked. These are isolated tests that verify a specific piece of the puzzle. None of these tests verify the entire lot by themselves (that would be an integration test - also valuable but not in scope of what you asked about).

In the spirit of "don't write code unless you have a failing test"

As with all broad advice, there are cases where you shouldn't take it too literally.

Technically, the advice is possible. If we start from an empty main logic method:

  • It doesn't call the database dependency
  • It doesn't call any simulation dependencies, let alone the correct number of times
  • It doesn't call the logger dependency

However, this is fairly trivial logic and it is silly to mandate that the tests must invariably have been written before you even touched this logic.

Yeah, the advice to write (failing) tests first makes sense in a larger codebase where you're trying to introduce or extend logic. In a trivial logic snippet, the value derived from doing it precisely that way decreases and can become obstructive.
(note: I'm not a purist about any design principle and I abhor dogmatic thinking, TDD purists might disagree with this paragraph)

Of course one solution is just to not strictly follow TDD for this part, but if there's another solution I'm open to learning before I give up. :)

I think you're overly focusing on the whole "first write failing tests" part. The intention of that advice is not to state that if you do anything else first, that it therefore invalidates the TDD-ness of the codebase. The intention is to point out that you should properly design and back your code using tests, rather than slapping them on at the end of the development process as an afterthought.

Avoid dogmatic thinking. Understand the spirit of the advice rather than the letter of the law. There is no universal piece of advice that does not have an edge case.

1
  • This is already a fairly open-minded frame-challenge-ish answer, but I think it would benefit from taking yet another step back. The "unit"'s responsibility is to integrate the others - and usually that follows some rather trivial logic (or else it could be broken out to a more easily tested unit). Tests that integrate a lot of mocked dependencies are often high-maintenance and low value compared to basic integration tests, and the right answer might be not having unit tests for this. Feb 14, 2023 at 7:31
2

How to TDD the main function?

one solution is just to not strictly follow TDD for this part

For what it is worth, on my own authority this is precisely what I've given myself permission to do.


I conclude that there are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies -- C.A.R Hoare, 1980

If we've written code that "obviously" has no deficiencies, are we really getting a good return on our investment trying to create tests to run in our refactoring loops?

Consider this sketch of a main method for a command line dice game:

// Java
public static void main (String [] args) {
    somethingMoreComplicatedButTestable(
        System.in,
        System.out,
        new Random()
    );
}

Do you really think an automated mistake detector targeting this code is going to pay for itself?

I get paid for code that works, not for tests, so my philosophy is to test as little as possible to reach a given level of confidence -- Kent Beck, 2008

Here, the confidence comes from being disciplined about two rules

  • code that is hard to test must be too simple to break
  • code that is complicated must be easy to test in isolation

TDD is an awareness of the gap between decision and feedback during programming, and techniques to control that gap. -- Test Driven Development by Example

Simplicity is a valid technique for controlling that gap.


See also: Mark Seemann's answer from 2014

2

If you really want follow the "don't write code unless .." rule dogmatically, make your "main program" a class of its own, with injected dependencies for the 3 steps you scetched in your question, and mocks which are designed to allow verifying that the main processing functions were called:

    TestMain()
    {
         //arrange
         DatabaseReader dr = new DatabaseReaderMock();
         Simulator sim = new SimulatorMock();
         OutputWriter ow = new OutputWriterMock();

         //act
         MainProgram mainProgram = new MainProgram(dr,sim,ow);
         mainProgram.Run();

         //assert
         Assert(dr.ReadDataWasCalled());
         Assert(sim.SimulateWasCalled());
         Assert(ow.WriteOutputWasCalled());
     }

This unit test design expects the MainProgram class to look somehow like this:

class MainProgram
{
     // DatabaseReader, Simulator  and OutputWriter are just interfaces
     MainProgram(DatabaseReader dr, Simulator sim, OutputWriter ow)
     {
         _dr=dr;
         _sim=sim;
         _ow=ow;
     }
     Run()
     {
         var data = _dr.ReadData();
         var output = _sim.Simulate(data);
         _ow.WriteOutput(output);
     }
}

You could also extend the test above and verify that Simulate was called with the output of "ReadData", and "WriteOutput" was called with the output of simulate, or that the three functions are called in that particular order (however, that may be at the borderline of expecting a specific implementation).

If you think this is worth the hassle, then go ahead. As an alternative, you may consider to rely on integration tests for this part of your system and do not write such a unit test.

Personally, I prefer pragmatism over dogmatism and would probably do the latter, as long as the MainProgram stays to be that simple. Moreover, whenever possible, I would try to keep the dataflow simple by a clear separation between processing components (DatabaseReader, Simulator, OutputWriter) and components which just orchestrate them (MainProgram).

IMHO "TDD by the book" rarely pays off for purely orchestrating components, but YMMV.

2

How to apply TDD to very high level functionality?

TDD doesn't care what level you're at. Levels are in our heads.

I want to apply TDD to the whole development process.

Then start with tests. Tests that mean something.

this is a rough outline of how the program will run:

  1. Read data from a database.
  2. Loop a bunch of times simulating the desired behavior.
  3. Write the output somewhere.

None of this is worth testing. Testing this only accomplishes locking down the structure of the program. Unless you have a product owner who says, "make sure you loop" stop distracting yourself from useful tests.

I completely reject this example because it's forcing us to look at this the wrong way. Let's say your program calculates π. That has a high level right? A good high level test tells you if you got the first 10 digits correct. Oh but it must also output a timestamp so we know when we calculated this. That's another test. Oh and a duration so we know how fast we calculated it. Yet another test. Do these other requirements keep our first test from being a high level test? No. It's simply a test that's about something important to us. It's not an intermediate part of something else. But it's also not the only thing going on.

What puts these tests on top is that they are independent of each other. You can remove any of them (and the code they're testing) and everything else just works.

TDD is more than happy to let you write up any of these tests before writing any code to actually do them. The tests don't need to know if the methods that satisfy them get their data from a DB, loop, or even write to the console. If you write tests that know that stuff you're insisting that that's how the methods must work. Where did you get that requirement from?

How would one generally approach this kind of situation?

The key thing I've done here is to not break down the steps of calculating π. Rather I've identified required behaviors that can be tested and satisfied independently.

Hey wait, how is the duration test independent of the calculate π test?

By writing it so that it doesn't care if it's timing the calculation or any other method.

But that's mocking!

Yes but it's mocking to make the tests requirement independent. Not because we crossed some meaningless arbitrary boundary like a class.

There is a school of thought that wants every class to be a unit surrounded by tests. I've worked in shops that follow this. I fully recommend doing that if you're goal is to create code that can't be changed without enormous pain.

If not then make your unit the functional core, not each class. Mock to keep your tests independent of each other. Mock to speed things up so you don't sit around waiting for the DB, file system, or network. Don't mock just because it's mockable.

You don't have to take my word for it. Fowler wrote about what he called solitary or sociable tests. His style is the sociable tests.

1

I think the point is being missed here.

You have two questions in fact:

  • One is how to test a function/class.
  • The other is what a test means in TDD

Testing a function/class should already be quite well practiced by you based on the question.

The answer is dependency injection.

The problem is that your main function assumes to know more than it should. It does not know every situation in which it might be created, same as any other class/function.

The solution is to humble it. It should accept those dependencies somehow, be it through a constructor, a property (which has some default), or an optional parameter.

This will make the program even more capable, as it is now much more configurable and testable.

If for some reason the test is complex to setup, then the function/class needs to be broken down. After all you are essentially asking anyone who uses your function/class to know all of this information in order to use it well. Most of the time that user is yourself.


As for the second question (which you didn't state but is lurking back there)...

English is tricky in that the meaning of a word can be broad or even change in a different context or usage. 'Test' is one such word.

The problem is that there are several kinds of test:

  • Aspirational Tests: These test describe something to be desired. In short a kind of Specification.
  • Adversarial Tests: These tests probe for weaknesses and exploitable gaps. In short evidence of safety/security.
  • Affirmational Tests: These tests pin down something that exists. In short evidence of organic code.

In TDD the goal is to write Aspirational tests.

The best time to write an aspirational test is when the code doesn't exist, or in a pinch without looking at the code (and you've had it unload from your own memory).

The worst time to write such tests is when the code exists. The temptation is very high to instead write an Affirmational, or an Adversarial test. Those tests have their benefits, but they won't deliver the thinking that an Aspirational test does.

An Aspirational test challenges you to simplify your tests. The simpler the tests:

  • the fewer prerequistes/knowledge needed to use the code effectively,
  • the more likely the code can be reused,
  • the easier it is to change the aspired for behaviour.
  • the easier it is to burn the implementation down and rebuild it,
  • the easier it is for another programmer (or your future self) to understand and maintain the code

Similarly this principal can be applied at higher levels of modularity under different names. eg: BDD, ATDD.

-1

I am not a big fan of mocking. It is useful in certain very special scenarios, but if you need a mocking framework, in my opinion you are doing something wrong.

Certain things need to be mocked because they cannot (should not...) be done in a unit test. For example accessing the file system to write output files. Or logging into the real production system, or even the real development system.

But your own logic should never be mocked away. Because in the end, you have thousands of unit tests, and not one of them actually tests whether your program is working! Every one is testing a tiny bit of functionality packed into a huge mass of mocking. So thousands of test could be green but when actually put together without mocks, it still fails. At least the possibility exists, and the whole point of tests is to make sure there is no such possibility.

Things you need to modify (I'm not saying "mock", because you can just switch out implementations, no need for a mocking framework):

  • Logging needs to log into a container that is accessible and constrained to that one test.
  • Data input needs to come from a source in that test. For example an in-memopry database unique to that test with data you decide on for each test.
  • Data output needs to go into a container that is accessible and constrained to that one test.

So your test should be your actual program, reading from an in-memory database, writing to an in-memory filesystem.

For the first test, it's an empty database. I don't know what that yields, maybe a file with no rows and just the column headers. Assert that. Done. Your first test is written and fails. Make it compile, build and light up green. Then add more.

The point is, do not waste 90% of your testing on building fancy mocks. Test real things. Test things that need testing, the mocking framework doesn't.

Next test might be with exactly one value in the database. That might yield a file of one row. And then your go to more complicated input and edge cases.

If you cannot build a public input for your test case, your program does not need that test case, or that functionality.

As an added bonus, if you test your programs public interface, you can refactor your heart out and you do not need to refactor your tests. Because they do what they were designed to do: test whether your program still works, not whether you have used those exact functions in that exact order.

To summarize:

  • Test reality. Not your mocking framework.
  • Test whether your program works, not what it does internally.
  • Start with the simplest case: no data.
  • Then branch out into all the test cases.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.