Should snapshot tests compare against stored test data or data from a database?

Question

I have a method GetReportAsync that takes one XML and generates another:

public async Task<string> GetReportAsync(string id)
{
    //  Get Order.xml from a file database
    var order = await _dataService.GetOrderAsync(id);

    var report = _reportGenerator.WriteReport(order);

    var xml = SerializeToXml(report);
    
    return xml;
}

As an automated testing tool, I write XMLUnit.NET snapshot tests that assert that an order will match a certain reference XML document, a stored snapshot.

[TestMethod]
[DataRow("12345")]
[DataRow("67890")]
public async Task Snapshots_Should_Match(string id)
{
    var actual = await _testClass.GetReportAsync(id);

    var expected = Input.FromFile($@"Snapshots\__snapshots__\{id}.xml");

    var diff = DiffBuilder
        .Compare(expected)
        .WithTest(actual)
        .Build();

    Assert.IsFalse(diff.HasDifferences());
}

Now, I realize that the Order.xml file can be retrieved in two ways:

• Alternative A: Store a number of Order.xml files in the project and implement _dataService.GetOrderAsync to read from these files.
• Alternative B: Get the actual Order.xml from a test (or production) database, just like in the real implementation.

Alternative A will assert that the method works given the order files in their exact state from when this snapshot test was written. However, I struggle to see the reason for such a guarantee.

Alternative B will give me tests that fail if the data service for some reason changes its response, possibly because of a non-backwards compatible change they introduce. To me this seems to give me more value than alternative A. However I do see that testing against a real database would possibly break some fundamental rules.

Furthermore, since the method is not writing to the database, then why not test against the production database instead of a test database?

Answer 1

My thought is you should use the alternative B.

As I can see if you have to change the code to run the alternative A, such behavior is not correct.

Also, alternative B seems mostly like a snapshot test because it preserves a system functionality.