How to create input data for unit tests for an API client?

Question

I am building the initial set of unit tests for my team's legacy API client system. We have been writing integration tests, but have no unit tests.

It's a Sinatra server that accepts requests from our web app and contacts the third-party APIs. Sometimes it returns the result to the web app.

                                 1                     2
(Web app) ---> Server endpoints ---> API Client class ---> 3rd party API
          <---                  <---                  <---
                                                        3

This being a legacy system, I am thinking about writing characterization tests. I will test the following:

1. mock our API Client class methods, to check how the input data to the server endpoints gets processed and formatted when it reaches our API Client class methods.
2. Similarly mock (or inject) the call to the 3rd party API, to check how the input data to API Client class method gets processed and formatted when it reaches the 3rd party API call.
3. Return canned server responses, to check how our program responds and processes each response.

Since the tests will be characterization tests, I will be just using the output generated by the code run by the test runner.

I am not sure how to generate inputs. The payloads are pretty complex objects, which can have ~30 fields (parameters) for some endpoints.

I don't think it is realistic to test all possible combinations of input parameters. Here's my plan:

• For each test, create a test input that's as typical as possible using the real payload. Save it in a file, as a fixture.
• Identify some fields that are interesting, important, or have previously caused errors. Create test cases for these fields. Load the fixture, and in each test case, overwrite a field in the payload with an extreme value, boundary value, or an illegal value, and making assertions with each payload. For example, if field A should be in range 5 < A < 15, with A = 5, A = 15, A = 4, A = 14, A = nil, etc.
• Run the test with failing expected values. Copy the actual output from the test runner, paste into expected values.

I am not sure if 1. this is the correct way to do characterization tests, and 2. this is a good way to create test input data. Am I overdoing it? Or doing it completely wrong?

Answer 1

Limit API Testing to the API Service, Not the Underlying System

There will be other points of view, but API testing should generally be limited to validating the existence of expected endpoints, authentication/authorization to the API itself (as opposed to auth against the back-end service behind the API), and validating the well-formedness of requests and responses. It's generally the wrong place to test application logic or complex objects that are more properly application client or server concerns.

Decomposing Your Question

You may get other answers that you like more, but in my personal opinion you're thinking about this problem wrong. You're actually asking three related questions, which need to be disentangled:

1. Can (or should) you write "unit" tests for an API server?
2. Should you use fixtures or factories?
3. How complete should your fixtures/factories be?

Everything in programming is a trade-off, so there's no universally correct answer here. But in the general case:

1. Only for behaviors unique to the API server.
2. Factories whenever possible (but note "trade-offs" as described above and below).
3. As incomplete as you can make them while still testing the behavior you're trying to validate.

Testing APIs

There are exceptions to every rule, but in my professional experience a well-written API is an interface or abstraction that sits on top of an actual implementation. In practice, that means it's certainly useful to test whether an API service accepts well-formed inputs, rejects poorly-formed inputs, and successfully passes/receives valid inputs to the service that the API exposes to a client.

So, why am I pointing this out? Because testing anything other than API message validation and forwarding is more likely to be testable on the application server (rather than the API service) at the unit level. If you want to include your API server in your end-to-end systems testing (and you generally should) that's fine; but don't try to bolt a ton of low-level tests that are really client or application server concerns onto an API service. If nothing else, you will find yourself asking questions like "How do I test low-level server stuff at the API interface level?" as you are. If it's painful to do, then that's usually an indication that you're doing the wrong thing.

Fixtures vs. Factories

There is a place in the world for both. There's no universally-correct answer, but in your case you're overlooking that "complex objects" require complex fixtures, and while it often seems easier to knock together a fixture object, you create technical debt whenever those complex objects change (as they do) as the application evolves. Fixtures are sometimes faster to create, and can be very useful in testing legacy applications (especially those with fewer testing seams), but they are often brittle and tightly coupled with the current implementation.

How Complete Should Your Fixtures/Factories Be?

Again, it depends. However, as with most testing, a generic rule of thumb is to focus most of your testing time and energy on things that are likely to break, have broken in the past, or represent boundary conditions that can get squirrely.

If you take my advice to do your unit testing against the client or server, rather than the API, then the answer is to test the bare minimum required for that unit. Are you testing authentication and authorization? Then your fixture or factory method should be testing just enough to validate that some unit correctly authenticates or authorizes, rather than implicitly testing the parsing or handling of complex message-passing with large composite (or worse yet, large monolithic) objects.

Your API service should not be in the business of handling most of these things directly, so it's the wrong place to be testing most application-level objects. If you find yourself testing anything other than elements unique to the API server, you are probably testing the wrong thing against the wrong part of the system.

Compose Larger Objects from Small, Independent Test Objects

As a special note, it's worth pointing out that building up complex objects through composition from independent factories is inherently useful because:

1. It often decouples your factories from one another, allowing you to keep your testing focused.
2. It acts as a form of executable documentation for your mega-widgets and uber-objects that are very hard to replicate in comments or written documentation.
3. It can expose when factories or the objects they generate know too much about one another, have too many external dependencies, or are otherwise in need of refactoring.
4. It makes debugging a lot easier, especially when there are small changes that would otherwise get buried inside a large uber-object.

Factories will often take more time to build than fixtures, especially if you didn't follow a solid TDD/BDD pattern during development in the first place. However, composing from smaller fixtures or factories will often help you avoid large chunks of tech debt when small parts of your application change.