2

Let's say I write a parser. It takes an argument that is a String and does something with that. This String can be very long so keeping it in a test class can dirty my code. I think that it would be better to keep this String in a file and read it during unit test.

I have doubts if it is still unit test or maybe integration test. I think that it is a unit test because still I test only my parser, not reading data from file. But I cannot find a source that can confirm my assumption. Can you tell me what type of test it is? It would be also great to prove this with some resource or something.

6
  • 1
    No consensus will ever emerge on this. Besides, it's mostly irrelevant: reading a file is part of the setup of the test, not the test itself - you could read files for both types of test, and for all other kinds of tests that exist on this planet.
    – user44761
    Sep 7 '16 at 10:24
  • 2
    Does it matter if it is an integration test or a unit test? Is it fundamentally different having "the data to test" in a file, or in the test, as a string? I'm happy to say it's a test, but I really don't care if it's a "unit" or "integration" test, it's necessary.
    – Vatine
    Sep 7 '16 at 10:33
  • Does it matter if it does matter? I asked because I want to know answer, thats it.
    – tdudzik
    Sep 7 '16 at 10:53
  • 1
    @tdudzik: the words "does it matter" are just a way of telling you "there is no broad consensus where to draw the line exactly, not even among experts, and you can call it whatever you like as long as you don't have a compelling reason why in your specific context you need to distinguish these terms in a specific way".
    – Doc Brown
    Sep 7 '16 at 10:59
  • If you are working in a CI environment it can matter since only unit tests are generally run on every check-in.
    – Robbie Dee
    Sep 7 '16 at 12:46
4

What makes a test an integration or a system test is the fact that the method under test relies on other parts of the system.

Imagine the following process:

  • The data is loaded from the database as an enumeration,
  • A method walks through the enumeration and aggregates the elements some way,
  • The result of the aggregation is stored back in the database.

When unit testing the method which aggregates the elements, you can't simply make it rely on the database calls: this wouldn't be a unit test because it has external dependencies; a mistake in the database code would fail the test, even if the method itself is correct. In other words, you can't correlate the test failure with the incorrectness of the method:

  • A test may start failing even if the method haven't changed since the last pass.
  • A regression may not fail the test because it is compensated by a change in the external dependency.

So you'll create stubs and mocks for the data access logic, in order to isolate the method. You pass a given enumeration to the method, and you check the result. No database involved.

Now, where do you get the data for the enumeration doesn't really matter. It may come from code. Or from the database. Or from the flat file. This is part of the test, not the code under test. If the test actually calls an API to get the elements, it's still a unit test.

However, unit tests are usually short and simple, because it's easier to reason about them when they are short and simple. Moving data to a file or a database or an API makes the test [unnecessarily] complicated. Now, when you need to check the unit test, you need to look not in one, but two locations. So while those are still unit tests, they can quickly become a maintenance nightmare. The second issue is that a failure in the external data source may lead to a failed unit test, unexpectedly.

  • Using a flat file may be OK (after all, source code is nothing more than a flat file too).

    However, I would find it strange that you need so much data for your unit test that you actually need to put it in a file.

    You must also be sure file system permissions don't get in the way of unit tests, and that the file won't be altered by mistake.

  • Using a database would be weird. How do you ensure the data is correct (if you fill this data in the test setup, then it doesn't make sense: you could pass it directly to the tested method, bypassing the database)? What about methods executing in parallel?

    Moreover, ensuring database is installed correctly and works is not an easy task. Unit tests probably don't need that complexity.

  • Using an API is even weirder. You don't normally control the API, which means that it could randomly fail, failing your unit tests with no reason.

3

My opinion: if the input string is so long that keeping it within the test method as a literal seems dirty, then the test is doing too much for a unit test. It's testing the parser on a long, complex piece of real-world data. That's fine, but usually the role of an integration test.

I'd expect a unit test to test a single feature of the parser, or each feature/code path/derivation once. If the point is to test the parser with long input string, you could dynamically create a long string through repeating a suitable fragment in a loop.

The question you could ask is: if the parser fails on this long input, could that also make it fail on a shorter input? If yes, then finding and using that shorter input is usually a good idea because the makes the test both shorter and faster.

3

When speaking about automated tests, there is no broad consensus where to draw the line between unit tests and integration tests exactly, not even among experts. Surely there are cases where most experts would say "this is clearly a unit test", and other cases where most people would say "no, this is not a unit test anymore".

But in your example, I am sure you will find experts who would deny that testing a parser with a huge, complex input string is still a "unit test", even when that string is directly encoded in the source code of your test. Others would argue "as long as you do not have to provide mocks or stubs to your parser, it is a unit test".

So call it a unit test, if you prefer that, and an integration test, if that fits better into the terminology of your environment.

2
  • As I've outlined, you may need to make the distinction for CI environments...
    – Robbie Dee
    Sep 7 '16 at 12:51
  • @RobbieDee: well, IMHO this just confirms that - depending on his environment - the OP might pick the term which suits his needs best, and that the criteria for making the distinction is probably not if the test data is stored in a file.
    – Doc Brown
    Sep 7 '16 at 14:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.