5

I'm working on adding unit/integration tests to an existing project (Java/Spring Boot) and I've been investigating on how they are "separated" in order to cover the test cases and how to focus each type of tests based on what they are written for.

To put things simple, my two questions are:

  1. Is it a good strategy to cover all test cases (including edge cases, exception handling, etc.) with unit tests and cover just some of all the test cases (let's say common user workflows) with integration tests?

  2. How do you approach writing integration tests? What do you usually focus on when writing integration tests for the functionalities you develop in your programs? Do you cover only a couple of happy paths or do you cover the same cases as you do with unit tests?

The first question is to know if the conclusions I've come to in order to do what I need to do are acceptable and that the strategy I'm thinking on is also acceptable. The second question is to get to know a bit more of how other developers actually do it in real life.

6
  • What do you need to know in order to be certain when your boss asks “Does it work correctly?”? Test for that. Dec 21, 2023 at 23:42
  • 2
    @ThorbjørnRavnAndersen But is your boss willing to pay for you to be certain. Or do they just want to pay for confident or even balance of probabilities? Dec 22, 2023 at 1:17
  • @user1937198 Well, this is about mindset where instead of just following cargo culture, you actually prepare the data to answer a real question. Then comes the boring details of what that question should be in order to get a realistic answer in time for an acceptable price. You are with other words later in the process. Dec 22, 2023 at 9:24
  • @user1937198, I think one problem between bosses and programmers, is ideas about what level of reliability is sufficient. There is rarely anything in computing where "balance of probabilities" reliability is actually organisationally useful. Something higher than 99% for the system is a more reasonable norm, and there are often many layers and parts in such a system which when analysed separately must be very much more reliable than 99%.
    – Steve
    Jan 1 at 15:19
  • @Steve Yet we don't have a language to discuss levels of reliability. And there are contexts where "balance of probabilities" reliability within the local context is useful. If the application has fast rollback, and failure would be detected by that rollback mechanism, and the outage inflicted by the rollback is an acceptable part of the error budget, then "balance of probabilities" as to whether it will deploy successfully is perfectly acceptable. Jan 1 at 15:59

5 Answers 5

11

Unit tests should test the behavior of your core code. They should be deterministic, parallelizable, and fast.

Integration tests should test that the rest of the system, that you had to cut out of the unit test to make it be those things, is at least integrated to the core correctly. This isn't about behavior. It's about if things are plugged in.

This isn't how everyone separates these responsibilities. But it's how I do it. I'm one of those annoying ones who wont let you call it a unit test just because you run it with jUnit.

Be aware, this testing pattern can be defeated by putting business rules in the database.

7

Wow, so many questions! Let's take a step back for a moment.

Why do we write tests? Any tests, of any kind?

The team spends time writing tests in order to save time.

Suppose a bug report is filed, and we need to diagnose whether the defect is in module A, B, or C? (I'm being simplistic, sometimes there's greater subtlety to it than that.) If A & B have such good unit test coverage that we feel confident in their correctness, we can triage the issue by focusing our efforts on how C might interact with the buggy behavior.

Suppose a change to existing feature or addition of new feature motivates refactoring our code to accommodate recently discovered needs. How do I know my refactor didn't break existing functionality? Test coverage gives us the courage to wade in, make changes, and still be confident the old stuff will keep on ticking.

There's a thread that runs through these: coverage. You should measure your current test coverage, and let that play a factor in deciding which tests to write next.


Ok, having reflected for a moment on what we hope to get out of them, let's look at the process of writing tests.

If you're thinking about it before writing a new feature, I claim you can "get your tests for free!". That is, you know you're going to have to execute that line of target code you wrote for the new feature. You could do it interactively. Or you could write an automated test which exercises the target code and verifies some sensible result. This initially looks a lot like traditional "printf debugging", but once the target code is doing the Right Thing the test can lock in the observed result and you get your test "for free". Months down the road future maintenance engineers will thank you, when they have to refactor that code.

Back to measuring coverage. Not all target code in your repo was written in that way. So now you want to retrofit tests on top of it. Start with a measurement, e.g. $ pytest --cov *.py. Or use poor man's profiling: set a debugger breakpoint for the method of interest, run tests, and either we hit the breakpoint or we don't. If not, write a covering test, and verify the breakpoint triggers.

But wait, you say. How do I know whether to write a unit test or an integration test? It doesn't matter. Just arrange for an automated test to exercise the source line of interest. If the API offers convenient access to the guts of the code, letting you call a tiny helper directly, then great! You have a tiny unit test. If the API makes it "hard" to do that, you might wind up with a big integration test which takes seconds to run and which pushes several frames onto the stack by the time it hits that line of code. Sigh! You have an integration test. Score a win anyway. Then write a ticket to change the target code's Public API so it is easier to directly test, and prioritize it in a subsequent sprint.

If you can cover the Happy Path for much of the code you care about, you're probably ahead of the game. Folks can spend a lot of time trying to move from 90% coverage to 95% coverage, exercising all those error handlers. Usually you will see diminishing returns from such effort. The important lesson to learn is that the Public API should be designed with all consumers in mind, both target code functions and test code functions. You might be able to refactor, to improve testability. More likely you will learn what traps to avoid next time when you're implementing some new feature that we might possibly wish to test.


  1. Is it a good strategy ...

I claim the most important thing to do, for a codebase with near zero coverage, is to write a couple of integration tests that exercise a fair bit of target code, even if stack depth gets a bit deep. Any exercised line at least has an opportunity to throw fatal error in CI/CD, so if someone broke it by always dividing by zero we will know about it quickly.

Once your code is largely covered, you can come back and write more narrowly focused unit tests. Often you will need DI and similar refactors, changes to your Public API, to make the target code unit testable. Pay attention to where most bug reports are coming from, and let that prioritize your refactors.

  1. How do you approach writing integration tests?

Write a Happy Path test for some input data that a stakeholder cares about. If you have more time, maybe try to exercise error handlers. But often that effort will be better spent writing narrowly focused unit tests that exercise error handlers.


Integration tests can be written to verify documented failure modes, e.g. deliberately supplying wrong password and verifying the expected "login denied, retry" user flow occurs. But typically the effort spent writing and monitoring an integration test is better spent if it tests a single item in a requirements document. Think of it as a small slice of an E2E system test. (And in the end you always a manual tester to occasionally follow a written sequence of test steps, to verify that Selenium or other automated test technology isn't hiding something that prevents users from accomplishing their goals. For example, a CSS "whoops!" that gives tiny font or low contrast text could be enough render a component unusable, despite the DOM having the "same" rendered objects it had last month when users were happy.)

If you want to exhaustively verify that error handlers function properly, and that all edge case inputs are dealt with, an integration test is not the way to go. Prefer narrowly focused unit tests for that.

If your app wasn't written with that in mind, you might need DI refactors or other (breaking!) changes to the Public API you designed, so it can accommodate the "one more" requirement of being readily testable.

7
  • For integration tests which execute large amounts of code that are ancillary to the actual test, how do you measure the coverage of what is actually tested and checked by the tests, as opposed to merely run, and the test will ignore whether the code gives a relevant answer at all? You can have 90%+ coverage, with a test suite that tests very few of the possible scenarios and errors if the application executed is large enough. Dec 29, 2023 at 18:03
  • @user1937198, I think I'm not yet buying the premise of the question. Suppose my integration test verifies that SELECT COUNT(*) ... WHERE ... returns more than a thousand matched rows; if N > 1000 we get a Green bar. It ran libc strcpy() code which is, maybe?, "ancillary". Also the kernel's socket read() code. Also some postgres client driver code, network code, backend code. But if any of those fail, e.g. due to mismatched glibc versions, I want to know about it, and I will get a Red bar. I measure Happy Path coverage of python source lines, not coverage within those infrastructure libraries.
    – J_H
    Dec 29, 2023 at 19:13
  • Take an example of reading a 1000 line file with 100 char fixed length lines. Each line contains 10 fields. I have a bug that reverses field one and 3. I write an integration test that checks it reads 1000 lines. According to my coverage I am covering the code that parses the fields from the lines. But because its not causing an explicit error, and not causing an issue that the test checks for, the test will still pass. Thats the kind of issue I'm talking about. Dec 30, 2023 at 1:09
  • 1
    @user1937198, I for one don't believe "coverage" (in the meaningful sense that all possible states, and state-paths, are considered correct - not just that every line of source is invoked by some test) can be "measured" at all. Tests are basically a kind of meta-program that tests the main program against an additional set of constraints. The reasonably necessary amount of testing applied is largely a product of experience of software in use - there's hardly any relevant "measurement", except for the cost of developer labour against the overall financial strength of the employer.
    – Steve
    Jan 1 at 15:52
  • 1
    @user1937198, very little, in my view. It tells you what area of source code cannot possibly be covered by an effective test (because it isn't reached by any test), but it is no indication of the effectiveness of any testing code that does reach those areas. On old code, I think it's certainly a mistake to aim first for high "source-code coverage", because retrofitted tests are more likely to be sloppy and miss subtleties anyway, and if they are devised at once all over a large (multi-person, multi-month) codebase, then developers who start carefully are likely to quickly become worn out.
    – Steve
    Jan 1 at 16:59
2

Integration tests cover an actual service call(s) to one or more dependencies. For example, it could be a database call, dropping a message to a queue, calling a 3rd party service, and/or calling an internal service.

It should be a real call with real data. What does it cover?

If the integration tests fail it could cover the possible failures:

  • Infrastructure related
  • Data related

Once the application is deployed, it can be a good idea to run a few integration tests to make sure everything is configured properly, and data is flowing into and out of the system. Maybe someone forgot a firewall rule or misspelled a database connection string. Sometimes this is called smoke testing but in essence it is an integration test. This is the first coverage point.

The second coverage point is to ensure that your existing service calls are returning/sending the expected data. For example, say your application calls a third-party service called https://thirdpartyservice/Foo which is a GET.

The integration test response should be:

{
    "name": "Bar",
    "age": 34
}

Let's say you don't get that back but instead get back:

{
    "lastName": "Bar",
    "age": 34
}

So, now it has failed. These can alert one to unexpected changes in services, changes in functionality, etc. It can serve as early detection for issues when functionality is updated that is beyond your control. Now this example is fairly simple with one of the attributes being returned having changed but hopefully it illustrates the point. It would still require a code change on your part to reflect the change and possibly updates to unit tests as well.

One shouldn't have a lot of integration tests, just a few to cover basic smoke testing and data interactions. Also, for CI/CD concerns, integration testing could be run for every deployment, but not every build. Probably best case is on demand as needed. It could be a post deployment step as well. Sometimes, the smoke testing is built into the application as part of a health check.

Unit tests cover your code and it's functionality. If those cases, the databases, external services, etc. are mocked or stubbed out because the tests are for your logic, not the dependency(s). Many times, if a dependency changes, one will need to update mocks/stubs to reflect the change in data from the dependent service.

0

Generally integration tests take much longer to run than unit tests, and it's harder to find problems that integration tests expose becacause they cover so much code. So yes, it's generally better to cover everything possible that you can with unit tests, and use integration tests only to cover things to difficult to cover with unit tests and, of course, test that units are plugged together correctly.

My general approach when writing a new program (or adding tests to an existing program with few to no tests) is to write an initial integration test that covers "end to end" use. For a command line program, for example, it would actually run the program with various parameters and inputs and examine the outputs.

Then, as I continue to build, modify and refactor the program, I try to add unit tests rather than additional integration test functionality, and also move what's currently into integration tests into unit tests. How much you can do in unit tests rather than integration tests depends a lot on the maturity of your code: when you have e.g. no parser at all for the input data, or a quick hack of a parser that covers just a few cases, it's easier to use an integration test to run data through the entire program. As the structure of such a parser becomes clear, it becomes easier and easier to test smaller and smaller parts of the parser with unit tests, and the risk of major structural changes requring a complete rework of the unit tests for that area lessens.

0

Usually in a request processing pipeline, there is an "IO - core logic - IO" sandwich. The IO can read/write from/to UI/DB/external service. The "core logic" part of the pipeline, devoid of any IO, should be separated into a (possibly "pure") function/method.

Then if only the "core logic" is tested, it would be a unit test.
But when the full pipeline is tested, with all IO operations, it would be an integration test.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.