1

I have a method GetReportAsync that takes one XML and generates another:

public async Task<string> GetReportAsync(string id)
{
    //  Get Order.xml from a file database
    var order = await _dataService.GetOrderAsync(id);

    var report = _reportGenerator.WriteReport(order);

    var xml = SerializeToXml(report);
    
    return xml;
}

As an automated testing tool, I write XMLUnit.NET snapshot tests that assert that an order will match a certain reference XML document, a stored snapshot.

[TestMethod]
[DataRow("12345")]
[DataRow("67890")]
public async Task Snapshots_Should_Match(string id)
{
    var actual = await _testClass.GetReportAsync(id);

    var expected = Input.FromFile($@"Snapshots\__snapshots__\{id}.xml");

    var diff = DiffBuilder
        .Compare(expected)
        .WithTest(actual)
        .Build();

    Assert.IsFalse(diff.HasDifferences());
}

Now, I realize that the Order.xml file can be retrieved in two ways:

  • Alternative A: Store a number of Order.xml files in the project and implement _dataService.GetOrderAsync to read from these files.
  • Alternative B: Get the actual Order.xml from a test (or production) database, just like in the real implementation.

Alternative A will assert that the method works given the order files in their exact state from when this snapshot test was written. However, I struggle to see the reason for such a guarantee.

Alternative B will give me tests that fail if the data service for some reason changes its response, possibly because of a non-backwards compatible change they introduce. To me this seems to give me more value than alternative A. However I do see that testing against a real database would possibly break some fundamental rules.

Furthermore, since the method is not writing to the database, then why not test against the production database instead of a test database?

  • 3
    ...then why not test against the production database instead of a test database? -- For several reasons. I think you can probably discard this as a possible option. If you want to exercise a real database, create one solely for testing purposes. – Robert Harvey Jan 4 at 15:33
  • @RobertHarvey Yes, I understand that, but what are your thoughts about alternative A and B? – Kristoffer Jälén Jan 5 at 9:23
  • 1
    It's the classic tradeoff between unit tests (essentially alternative A) and integration tests (essentially alternative B). – Robert Harvey Jan 5 at 12:51
  • What are you testing here? DiffBuilder? GetReportAsync? Both? You can test DiffBuilder without data sources, just with stubbs. – Laiv Jan 8 at 13:08
0

My thought is you should use the alternative B.

As I can see if you have to change the code to run the alternative A, such behavior is not correct.

Also, alternative B seems mostly like a snapshot test because it preserves a system functionality.

Not the answer you're looking for? Browse other questions tagged or ask your own question.