Unit Testing with massive lookup tables?

Question

Our system is structured in such a way that we get a lot of key info for our calculations and other such logic from lookup type tables. Examples would be all kinds of different rates (like interest rates or contribution rates), dates (like effective dates), and all of kinds of different misc info.

Why did they decide to structure everything like this? Because some of this info changes quite often. For example some of our rates changes yearly. They wanted to try to minimize code changes. The hope was just that the lookup tables would change and the code would just work (no code changes).

Unfortunately, I think its going to make unit testing challenging. Some of the logic might make 100+ different lookups. While i can definitely make a mockable object that returns our rates there is going to be considerable setup. I think its either that or i have to end up using integration tests (and hitting that database). Am i right or is there a better way? Any suggestions?

Edit:
Sorry for the delayed response but i was trying to soak everything in while at the same thing juggling many other things. I also wanted to try to work through the implementation and at the same time. I tried a variety of patterns to try to architect the solution to something i was happy with. I tried the visitor pattern which i wasn't happy with. In the end i ended up using the onion architecture. Was i happy with the results? Sort of. I think it is what it is. The lookup tables make it a lot more challenging.

Here is a small example (im using fakeiteasy) of setup code for the tests for a rate that changes yearly:

private void CreateStubsForCrsOS39Int()
{
    CreateMacIntStub(0, 1.00000m);
    CreateMacIntStub(1, 1.03000m);
    CreateMacIntStub(2, 1.06090m);
    CreateMacIntStub(3, 1.09273m);
    CreateMacIntStub(4, 1.12551m);
    CreateMacIntStub(5, 1.15928m);
    CreateMacIntStub(6, 1.19406m);
    CreateMacIntStub(7, 1.22988m);
    CreateMacIntStub(8, 1.26678m);
    CreateMacIntStub(9, 1.30478m);
    CreateMacIntStub(10, 1.34392m);
    CreateMacIntStub(11, 1.38424m);
    CreateMacIntStub(12, 1.42577m);
    CreateMacIntStub(13, 1.46854m);
    CreateMacIntStub(14, 1.51260m);
    CreateMacIntStub(15, 1.55798m);
    CreateMacIntStub(16, 1.60472m);
    CreateMacIntStub(17, 1.65286m);
    CreateMacIntStub(18, 1.70245m);
    CreateMacIntStub(19, 1.75352m);
    CreateMacIntStub(20, 1.80613m);
    CreateMacIntStub(21, 1.86031m);
    CreateMacIntStub(22, 1.91612m);
    CreateMacIntStub(23, 1.97360m);
    CreateMacIntStub(24, 2.03281m);
    CreateMacIntStub(25, 2.09379m);
    CreateMacIntStub(26, 2.15660m);
    CreateMacIntStub(27, 2.24286m);
    CreateMacIntStub(28, 2.28794m);
    CreateMacIntStub(29, 2.35658m);
    CreateMacIntStub(30, 2.42728m);
    CreateMacIntStub(31, 2.50010m);
    CreateMacIntStub(32, 2.57510m);
    CreateMacIntStub(33, 2.67810m);
    CreateMacIntStub(34, 2.78522m);
    CreateMacIntStub(35, 2.89663m);
    CreateMacIntStub(36, 3.01250m);
    CreateMacIntStub(37, 3.13300m);
    CreateMacIntStub(38, 3.25832m);
    CreateMacIntStub(39, 3.42124m);
    CreateMacIntStub(40, 3.59230m);
    CreateMacIntStub(41, 3.77192m);
    CreateMacIntStub(42, 3.96052m);
    CreateMacIntStub(43, 4.19815m);
    CreateMacIntStub(44, 4.45004m);
    CreateMacIntStub(45, 4.71704m);
    CreateMacIntStub(46, 5.00006m);
    CreateMacIntStub(47, 5.30006m);
    CreateMacIntStub(48, 5.61806m);
    CreateMacIntStub(49, 5.95514m);
    CreateMacIntStub(50, 6.31245m);
    CreateMacIntStub(51, 6.69120m);
    CreateMacIntStub(52, 7.09267m);
    CreateMacIntStub(53, 7.51823m);
    CreateMacIntStub(54, 7.96932m);
    CreateMacIntStub(55, 8.44748m);
    CreateMacIntStub(56, 8.95433m);
    CreateMacIntStub(57, 9.49159m);
    CreateMacIntStub(58, 10.06109m);
    CreateMacIntStub(59, 10.66476m);
    CreateMacIntStub(60, 11.30465m);
    CreateMacIntStub(61, 11.98293m);
    CreateMacIntStub(62, 12.70191m);
    CreateMacIntStub(63, 13.46402m);
    CreateMacIntStub(64, 14.27186m);
    CreateMacIntStub(65, 15.12817m);
    CreateMacIntStub(66, 16.03586m);
    CreateMacIntStub(67, 16.99801m);
    CreateMacIntStub(68, 18.01789m);
    CreateMacIntStub(69, 19.09896m);
    CreateMacIntStub(70, 20.24490m);
    CreateMacIntStub(71, 21.45959m);
    CreateMacIntStub(72, 22.74717m);
    CreateMacIntStub(73, 24.11200m);
    CreateMacIntStub(74, 25.55872m);
    CreateMacIntStub(75, 27.09224m);
    CreateMacIntStub(76, 28.71778m);

}

private void CreateMacIntStub(byte numberOfYears, decimal returnValue)
{
    A.CallTo(() => _macRateRepository.GetMacArIntFactor(numberOfYears)).Returns(returnValue);
}

Here is some setup code for a rate that can change at any point (it might be years before a new interest rate is introduced):

private void CreateStubForGenMbrRateTable()
{
    _rate = A.Fake<IRate>();
    A.CallTo(() => _rate.GetRateFigure(17, A<System.DateTime>.That.Matches(x => x < new System.DateTime(1971, 7, 1)))).Returns(1.030000000m);

    A.CallTo(() => _rate.GetRateFigure(17, 
        A<System.DateTime>.That.Matches(x => x < new System.DateTime(1977, 7, 1) && x >= new System.DateTime(1971,7,1)))).Returns(1.040000000m);

    A.CallTo(() => _rate.GetRateFigure(17,
        A<System.DateTime>.That.Matches(x => x < new System.DateTime(1981, 7, 1) && x >= new System.DateTime(1971, 7, 1)))).Returns(1.050000000m);
    A.CallTo(
        () => _rate.GetRateFigure(17, A<System.DateTime>.That.IsGreaterThan(new System.DateTime(1981, 6, 30).AddHours(23)))).Returns(1.060000000m);
}

Here is the constructor for one of my Domain Objects:

public abstract class OsEarnDetail: IOsCalcableDetail
{
    private readonly OsEarnDetailPoco _data;
    private readonly IOsMacRateRepository _macRates;
    private readonly IRate _rate;
    private const int RdRate = (int) TRSEnums.RateTypeConstants.ertRD;

    public OsEarnDetail(IOsMacRateRepository macRates,IRate rate, OsEarnDetailPoco data)
    {
        _macRates = macRates;
        _rate = rate;
        _data = data;
    }

So why don't i like it? Well existing tests will work but anyone adding any new test in the future will have to look through this setup code to make sure any new rates are added. I tried to make it as clear as possible by using the table name as part of the function name but i guess it is what it is :)

Answer 1

You can still write unit tests. What your question describes is a scenario in which you have some data sources that your code depends on. These data sources need to produce the same fake data across all of your tests. However, you don't want the clutter associated with setting up responses for every single test. What you need are test fakes

A test fake is an implementation of something that looks like a duck and quacks like a duck, but doesn't do anything but provide consistent responses for the purposes of testing.

---

In your case, you may have an IExchangeRateLookup interface and a production implementation of it

public interface IExchangeRateLookup
{
    float Find(Currency currency);
}

public class DatabaseExchangeRateLookup : IExchangeRateLookup
{
    public float Find(Currency currency)
    {
        return SomethingFromTheDatabase(currency);
    }
}

By depending on the interface in the code under test, you're able to pass in anything which implements it, including a fake

public class ExchangeRateLookupFake : IExchangeRateLookup
{
    private Dictionary<Currency, float> _lookup = new Dictionary<Currency, float>();

    public ExchangeRateLookupFake()
    {
        _lookup = IntialiseLookupWithFakeValues();
    }

    public float Find(Currency currency)
    {
        return _lookup[currency];
    }
}

Answer 2

The fact that:

Some of the logic might make 100+ different lookups.

is irrelevant in a context of unit testing. A unit tests focuses on a small part of the code, usually a method, and it is unlikely that a single method needs 100+ lookup tables (if it does, refactoring should be your top concern; testing comes after that). Unless you mean 100+ lookups in a loop to the same table, in which case, you're fine.

The complexity of adding stubs and mocks for those lookups shouldn't bother you either at a scale of a single unit test. Within the test, you will stub/mock only those of the lookups which are actually in use by the method. Not only won't you have a lot of them, but also those stubs or mocks will be very simple. For instance, they may return a single value, no matter what the method is looking for (as if an actual lookup was filled with the same number).

When the complexity will matter is when you'll have to test the business logic. 100+ lookups probably means thousands and thousands of different business cases to test (even outside lookups), which means thousands and thousands of unit tests.

Illustration

For example, in a context of an OLAP cube, you may have a method which relies on two cubes, one with two dimensions and one with five dimensions:

public class HelloWorld
{
    // Intentionally hardcoded cubes.
    private readonly OlapCube olapVersions = new VersionsOlapCube();
    private readonly OlapCube olapStatistics = new StatisticsOlapCube();

    ...

    public int Demo(...)
    {
        ...
        this.olapVersions.Find(a, b);
        ...
        this.olapStatistics.Find(c, d, 0, e, 0);
        ...
    }
}

As is, the method cannot be unit tested. The first step is to make it possible to replace OLAP cubes by stubs. One way to do it is through Dependency Injection.

public class HelloWorld
{
    // Notice the interface instead of a class.
    private readonly IOlapCube olapVersions;
    private readonly IOlapCube olapStatistics;

    // Constructor.
    public HelloWorld(
        IVersionsOlapCube olapVersions, IStatisticsOlapCube olapStatistics)
    {
    }

    ...

    public void Demo(...)
    {
        ...
        this.olapVersions.Find(a, b);
        ...
        this.olapStatistics.Find(c, d, 0, e, 0);
        ...
    }
}

Now a unit test can inject a stub like this:

class OlapCubeStub : IOlapCube
{
    public OlapValue Find(params int[] values)
    {
        return OlapValue.FromInt(1); // Constant value here.
    }
}

and used like that:

var helloWorld = new HelloWorld(new OlapCubeStub(), new OlapCubeStub());
var actual = helloWorld.Demo();
var expected = 9;
this.AssertEquals(expected, actual);