8

People say that "talking about TDD hardly works, if you want to convince someone to TDD, show them results". However, I'm already getting great results without TDD. Showing me that people who use TDD get good results won't be convincing, I want to see that people who write both TDD and not-TDD get better results with TDD.

Despite all of this, I'm interested in giving TDD a try. However I'm not convinced I will gain anything from this. If it does prove useful, I will try to push it to the rest of my team.

My main question is this: Would TDD serve any purpose for code, if I can already prove the code correctness?

Obviously, neither one is a silver bullet. Your proof might be wrong because you missed a detail, and your test might fail to spot a bug that you failed to test for. In the end, we're human, nobody can make 100% bug-free-code forever. We can only strive to get as close as possible.

However, would TDD actually save any time on code that had its correctness proven? i.e. code that, in the state machine that the code operates on, all valid possible states and their ranges are recognized by the developer, all are accounted for, and the code is designed in a whitelist-style error-checking that passes every exception to an upper handler to make sure nothing unexpected leaks -> without both displaying a (within-reason-)relevant message to the client and sending log notifications to an admin.

Answers with real-life examples would be better.

Some clarifications:

  • This question is not about whether you can prove code correctness or not. Lets assume by default that not all code can be proven correct within a reasonable timeframe, but that some pieces of code can be. For example, it's very easy to proven correctness of a FizzBuzz module. Not very easy for a cloud-based data syncing service.

  • Within this confine, the question asks the following: Start with the assumption that a codebase is divided into 2 parts: [I] parts that have been proven correct [II] parts that have not been proven correct, but manually tested to work.

  • I want to apply TDD practices to this codebase that did not have them until now. The question asks the following: should TDD be applied to every single module, or would it be enough to apply them to only modules that were not proven correct?

  • "Proven correct" means that you can consider this module completely functional-style, i.e., it does not rely on any global or outer state outside itself, and has entirely its own API for I/O that other modules that interact with it must follow. It is not possible to "break this module" by changing code outside the module, at worst you can misuse it and get formatted error messages returned to you.

  • Obviously, every rule has exceptions, compiler bugs in new compiler versions may introduce bugs to this module, but the same bugs could be introduced to tests that tested it and result in a false sense of safety from tests that no longer work as intended. The bottom line is that tests are not a magical solution, they're another layer of protection, and this question discusses the issue of whether this layer of protection is worth the effort in the specific case of a module that was proven correct (assume that it was indeed).

Improve this question
18
  • 10
    Proving "code correctness" might become harder that you think it is.
    – πάντα ῥεῖ
    Apr 2, 2018 at 17:56
  • 35
    While I enjoyed learning about your workplace and your work history, I almost stopped reading several times because well, sometimes what's really important is that, as a way of maximising your results, and, in order to keep your reader's attention so that they can, in an effort to strengthen your knowledge as well as the community's, help you, you must... get to the point. Please consider shortening your question to its most salient points.
    – MetaFight
    Apr 2, 2018 at 19:07
  • 10
    What you describe is not process of "proving correctness" of code. That kind of process is drastically different. Also, I find it hard to accept that you can build code in a way that can be "proved correct" just by looking at it. I saw plenty of code that seemed trivial and "correct", only to be totally crushed when put under robust automated test.
    – Euphoric
    Apr 2, 2018 at 19:44
  • 8
    "Beware of bugs in the above code; I have only proved it correct, not tried it." -Donald Knuth.
    – Neil
    Apr 3, 2018 at 13:01
  • 6
    Your question makes no sense. TDD means that the tests drive the development. In other words, you have no design, no architecture, no code, unless you have a test for it. So, how in the world are you "applying TDD to code that has been proven correct" when by the very definition of TDD, there is no code to prove correct?
    – Jörg W Mittag
    Apr 3, 2018 at 16:21

7 Answers 7

Reset to default
20

Yes.

Proofs are fine when they're available, but even at the best of times they only prove that a single bit of code will work as expected (for all inputs? accounting for interruptions in the middle of any operation? what about running out of memory? disk failure? network failure?).

What happens when it changes?

Tests are great because they serve as an implied contract about what the code should do. They provide some scaffolding so that your new intern can go in and make changes with some level of confidence. All via quick, clear results: pass or fail.

And frankly, I can coach an intern to write viable unit tests in a few months. I doubt that anyone on my team (myself included) could create proofs that guarantee anything meaningful for non-trivial code; let alone do it quickly and accurately.

Improve this answer
6
  • The same code that would be non-trivial to prove would also be non-trivial to test. A network failure or a disk failure can take a lot of forms, how can you be confident that your tests covered every single possible scenario? They will most likely cover every single scenario you could think of, but not necessarily every single scenario in real life. You can't just blindly assume every change will be non-breaking just because you have tests in place. It's just another protection layer. Tests are important, but they're not a silver bullet.
    – Kylee
    Apr 2, 2018 at 21:15
  • 6
    @Kylee - no offense, but you seem to be wildly underestimating the effort and skill necessary to do a proof of correctness (or wildly overestimating the effort and skill necessary to toss together some automated tests).
    – Telastyn
    Apr 2, 2018 at 21:37
  • Maybe you're right, and I most likely do overestimate the effort in putting together tests due to being inexperienced with it (honestly, it's less the "putting together tests" that I'm concerned about, and more about actually running the tests. For large & rapidly changing projects, that in itself is not a negligible time sink). Either way, this has nothing to do with the question. The question clearly states, assume that you have code that was already proven correct, is there still a point to write unit tests for it?
    – Kylee
    Apr 2, 2018 at 22:02
  • 1
    For large & rapidly changing projects the more prone to change the more necessary are the tests, since a changing code has many more chances to fail due to new bugs or unexpected behaviours, than code that barely change. It's a matter of probability. Even if it doesn't change often, after a while the knowledge obtained during the development could get lost or fall into oblivion. Tests are also materialised knowledge, that can cut down, significantly, the learning curve. Is coding tests time-consuming? Yes. Does it make more expensive the project? No, in the long run make it cheaper.
    – Laiv
    Apr 4, 2018 at 6:38
  • 3
    @Kylee Even if you could prove that a piece of code is correct, at best you would be in the situation that no one is allowed to change that code or add features to it; to do so would invalidate your proof. As already highlighted in this answer, tests allow you to change your code with confidence. Even if the person changing it is an inexperienced intern. By the way, even if some block of code is 100% correct does not mean you will never need to change it (e.g. to add a new feature needed by a customer or to satisfy some real world constraint not considered in your proof).
    – Brandin
    Apr 9, 2018 at 12:48
5

We don't know. We cannot answer your question.

While you spend lots of time explaining that process you have now seems to work to everyone's satisfaction, you are telling us only small sliver of what is actually happening.

From my experience, what you are describing is extreme rarity and I'm skeptical that it is actually your process and approach to coding that is actually cause of low bug count in your applications. There might be many other factors that influence your applications and you are telling us nothing about those factors.

So we don't know, in face of not knowing your exact development environment and culture, if TDD will help you or not. And we can spend days discussing and arguing about it.

There is only one recommendation we can give you : try it out. Experiment. Learn it. I know you are trying to spend least amount of effort to decide, but that is not possible. If you really want to know if TDD will work in your context, only way to find out is to actually do TDD. If you actually learn it and apply it to your application, you can compare it with your non-TDD process. It might be that TDD actually has advantages and you decide to keep it. Or it can come out that TDD doesn't bring anything new and only slows you down. In which case, you can fall back to your previous process.

Improve this answer
5

The main purpose of (unit) tests is safeguarding code, making sure it will not break unnoticed because of later changes. When the code is first written, it will get a lot of attention and it will be scrutinized. And you may have some superior system for that.

Six months later, when someone else is working on something seemingly unrelated, it may break and your super-duper code-correctness-prover will not notice it. An automatic test will.

Improve this answer
1
  • 1
    This is something that is far too often overlooked. Yes, there is pain to go through to make sure the code has unit tests now. But this will pay for itself many times over when a junior dev does a simple change and unit tests immediately flag up that they have broken something else. I've been the junior dev decades ago who has broken something and the whole release has been delayed. While my colleagues were very tolerant at the time and had my back when managers started jumping up and down, the whole scenario could, in hindsight, have been avoided if unit tests had been in place.
    – Robbie Dee
    Apr 5, 2018 at 8:04
5

I want to apply TDD practices to this codebase that did not have them until now.

This is the hardest way to learn TDD. The later you test, the more it costs to write tests and the less you get out of writing them.

I'm not saying it's impossible to retrofit tests into an existing code base. I'm saying doing so isn't likely to make anyone into a TDD believer. This is hard work.

It's actually best to practice TDD the first time on something new and at home. That way you learn the real rhythm. Do this right and you'll find it addictive.

The question asks the following: should TDD be applied to every single module,

That is structural thinking. You shouldn't say things like test every function, or class, or module. Those boundaries are not important to testing and they should be able to change anyway. TDD is about establishing a testable behavioral need and not caring how it's satisfied. If it wasn't we couldn't refactor.

or would it be enough to apply them to only modules that were not proven correct?

It's enough to apply them where you find a need for them. I'd start with new code. You'll get much more back from testing early than from late. Don't do this at work until you've practiced enough to master it at home.

When you've shown TDD is effective with the new code at work and feel confident enough to take on the old code I'd start with the proven code. The reason why is because you'll be able to see right away if the tests you're writing are taking the code in a good direction.

My main question is this: Would TDD serve any purpose for code, if I can already prove the code correctness?

Tests don't just prove correctness. They show intent. They show what is needed. They point out a path to change. A good test says there are several ways to write this code and get what you want. They help new coders see what they can do without breaking everything.

Only once you have that down should you wander into the unproven code.

A warning against zealots: You sound like you've achieved success and so will be unlikely to jump in headfirst. But others looking to prove themselves will not be so reserved. TDD can be overdone. It's amazingly easy to create a suite of tests that actually hurts refactoring because they lock down trival and meaningless stuff. How does this happen? Because people looking to show off tests just write tests and never refactor. Solution? Make them refactor. Make them deal with feature changes. The sooner the better. That will show you the useless tests quickly. You prove flexibility by flexing.

A warning against structural categorizing: Some people will insist that a class is a unit. Some will call any test with two classes an integration test. Some will insist that you can't cross boundary x and call it a unit test. Rather than care about any of that I advise you to care about how your test behaves. Can it run in a fraction of a second? Can it be run in parallel with other tests (side effect free)? Can it be run without starting up or editing other things to satisfy dependencies and preconditions? I put these considerations ahead of if it talks to a DB, file system, or network. Why? Because these last three are only problems because they cause the other problems. Group your tests together based on how you can expect them to behave. Not the boundaries they happen to cross. Then you know what you can expect each test suite to do.

I've seen people say they don't want to use TDD because it would have too much overhead, and TDD supporters defend it by saying that once you get used to write TDD all the time there isn't much overhead.

That question already has answers here.

Improve this answer
1

Test Driven Development is more about prototyping and brainstorming an API, than testing. The tests created are often poor quality and eventually have to be thrown out. The main advantage of TDD is determining how an API will be used, before writing the API implementation. This advantage can also be obtained in other ways, for example by writing API documentation before the implementation.

Correctness proofs are always more valuable than tests. Tests don't prove anything. However, in order to use correctness proofs productively, it helps to have an automated proof checker, and you will need to work using contracts of some sort (design by contract or contract based design).

In the past, when working on critical sections of code, I would attempt manual correctness proofs. Even informal proofs are more valuable than any automated tests. But you still need the tests, unless you can automate your proofs, as people will break your code in the future.

Automated tests do not imply TDD.

Improve this answer
7
  • Brainstorming an API and design by contract is already how I approach problems, so I'd like to say that I like your answer, but other sources say "In test-driven development, each new feature begins with writing a test". You said things that I agree with, but did not convince me to use TDD approach. "Automated tests do not imply TDD", ok, but then, what does TDD imply, as opposed to simply following general best practices?
    – Kylee
    Apr 2, 2018 at 18:32
  • 1
    While I agree that TDD often involves brainstorming an API, IMO, this is sometimes a bad thing. The API shouldn't be designed (necessarily) for testability, it should be designed to be used smoothly by your clients, and to make sense to the classes / code involved. All too often I've seen a test writer say "lets add String getSomeValue() here so we can test it" when that makes no sense for the overall design. Sure, you could remove that function later, but, in my experience, that is rare.
    – user949300
    Apr 2, 2018 at 18:43
  • 1
    @user949300, There is a large overlap between designing a testable API and one designed to be used smoothly by your clients. Adding unnecessary code for the sake of the test shows you have a bad design. All too often, API writers forget about debugging what is going wrong with an API. Writing something testable AND useful to the user forces you to think about those things... without leaking implementation details into your interface.
    – Berin Loritsch
    Apr 2, 2018 at 18:46
  • @user949300 The number one complaint about TDD is probably the way the API is modified for "testability", in the unit sense, such that things that are usually encapsulated are exposed. The number two complaint is probably that it doesn't scale well over time.
    – Frank Hileman
    Apr 2, 2018 at 21:17
  • @Kylee Automated tests are older than the buzzword "TDD". The difference is that automated tests are simply what you think should be tested -- there is no dogma, nor specific order in which you write them. Nor is there any emphasis on unit vs integration tests.
    – Frank Hileman
    Apr 2, 2018 at 21:20
0

A) You reading the code and convincing yourself that it's correct isn't remotely close to proving it's correct. Otherwise why write tests at all?

B) When you change the code you want to have tests run that demonstrate the code is still correct or not.

Improve this answer
2
  • this seems to merely repeat points made (and much better explained) in top answer that was posted few weeks earlier
    – gnat
    Apr 24, 2018 at 7:51
  • @gnat It's a more succinct response for something that doesn't need a novel written about it. Please try to contribute something here.
    – Josh
    May 1, 2018 at 19:28
-1

I will caveat by saying that once you are used to using TDD effectively, it will save you time in the end-game. It takes practice to learn how to use TDD effectively, and it doesn't help when you are under a time crunch. When learning how to best make use of it, I recommend starting on a personal project where you have more leeway and less schedule pressure.

You'll find that your initial progress is slower while you are experimenting more and getting your API written. Over time, your progress will be quicker as your new tests start passing without changing code, and you have a very stable base to build from. In the late game, code that is not built using TDD requires you to spend a lot more time in the debugger as you try to figure out what is going wrong than should be necessary. You also run greater risk of breaking something that used to be working with new changes. Assess the effectiveness of TDD vs. not using it by total time to completion.

That said, TDD is not the only game in town. You can use BDD which uses a standard way of expressing the behavior of a full-stack application, and assess the correctness of the API from there.

Your whole argument hinges on "proving code correctness", so you need something that defines code correctness. If you aren't using an automated tool to define what "correct" means, then the definition is very subjective. If your definition of correct is based on the consensus of your peers, that can change on any given day. Your definition of correct needs to be concrete and verifiable, which also means it should be able to be evaluated by a tool. Why not use one?

The #1 win from using automated testing of any sort, is that you can verify your code remains correct even when OS patches are applied quickly and efficiently. Run your suite to make sure everything is passing, then apply the patch and run the suite again. Even better, make it part of your automated build infrastructure. Now you can verify your code remains correct after merging code from multiple developers.

My experience using TDD has led me to the following conclusions:

  • It's great for new code, difficult for changing legacy systems
  • You still have to know what you are trying to accomplish (i.e. have a plan)
  • Slow to start, but saves time later
  • Forces you to think about how to validate correctness and debugging from a user perspective

My experience using BDD has led me to the following conclusions:

  • It works for both legacy and new code
  • It validates the whole stack, and defines the specification
  • Slower to get up and running (helps to have someone who knows the toolset)
  • Fewer behaviors need to be defined than unit tests

Definition of Correct: Your code complies with requirements. This is best verified with BDD, which provides a means of expressing those requirements in a human readable fashion and verifying them at run time.

I am not talking about correctness in terms of mathematical proofs, which is not possible. And I am tired of having that argument.

Improve this answer
7
  • A wonderful post that does address an issue that made me consider TDD to begin with: the current definition of "code correctness" is indeed consensus of the peers. My concern is that as the team changes in the future, this method would stop working. But, how TDD would solve this? While old tests may save new team members from breaking old functionality easily, they may still write incomplete tests for future features, that would still lead into problems. In the end, both methods rely on trusting your team. I've not heard of BDD before, so going to check that out as well. Thanks.
    – Kylee
    Apr 2, 2018 at 19:23
  • 2
    "you can verify your code remains correct" Nope. Not even close. You can at most claim that the test that you've run still passes.
    – Bent
    Apr 2, 2018 at 20:48
  • @Kylee, BDD addresses that concern better. BDD has you write a spec that is verifiable. The spec is written in your natural language with means of creating hooks to actual test code that verifies the spec. That bridges two concerns, communicating the actual requirements and enforcing them.
    – Berin Loritsch
    Apr 3, 2018 at 12:32
  • 1
    @Bent, Please don't argue about "correctness" in terms of mathematical proofs. That's not the topic of conversation or what I intended to convey. However, based on experience people who post comments like yours tend to have that in mind. You can absolutely verify that code complies with requirements. That is the working definition of correct that I am talking about.
    – Berin Loritsch
    Apr 3, 2018 at 12:34
  • The trouble is, you're driving to a definition of "correct", where the requirement is that the code passes the tests you've defined. Any other set of inputs is undefined behaviour, and the output is arbitrary. This doesn't really match what most users would consider to be the requirements.
    – Simon B
    Apr 4, 2018 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.