6

I've been coding as a career for about 2 years now but am just now writing my first "real" tests for a non-trivial application. I've been trying to follow best practices I've picked up from the internet here and there but I would like some feedback on whether I am heading in the right direction with what I am doing and what I can do to correct/improve my process.


The application is a headless API that will be seeing public (aka clients who want to integrate with our API, not totally public) consumption in a few months.

From top to bottom the stack and layers look like this:

  • RESTful API (JSON)
  • Oauth 2.0 authentication
  • (Mostly) CRUD controllers (PHP, Laravel 5)
  • Doctrine ORM for DAL
  • MySQL DB

I wanted to design my tests to be as simple but thorough as possible. However I don't have eternity and a day to write really fundamental test (unit tests I guess) for all controllers and all business logic so I decided to take advantage of laravel's built in mocking for routes and test all endpoints for our API.

My line of thinking is that, since we have standard data structures and formats for request/response (JSON, defined json schemas for objects, etc.) I could emulate requests knowing what the response should look like and compare the two.

EX. GET api/user/1 should return user 1 as a json object that I can compare for equality with the serialized json the actual object gives me.


So the actual testing looks like this (using PHPunit):

  1. On application start Laravel sets the app environment to "testing" which changes the default DB connection to a SQLite file
  2. On PHPunit boot Doctrine gets model metadata and creates tables for my domain in the DB, some extra scripts for extra-domain stuff (oauth tables) create those as well
  3. Alice, an object generator for doctrine, creates a decently diverse set of domain entities with random data from Faker and hydrates the SQLite DB. All created entities are returned and I store these in a global variable for later use.
  4. I make a copy of the hydrated, untouched DB to use in tests that involve write operations so I can reset the db after each test.
  5. Oauth clients get generated for users with different roles so I can test oauth scopes and permissions across different users for each test
  6. I use Laravel's route mocking to call each API endpoint and
  7. For read-only tests (GETs) I use the json serialized form of objects stored earlier (step 3) to test the equality of responses,
  8. For write tests I construct requests from our object json schemas, then build a real object that should equate it, make the request and then check the response for equality with the serialized form of the object (like step 7). After each write test the DB is overwritten with the original hydrated DB from step 4 for the next test.

So that's the gist of it. Using this approach has given me fantastic performance -- running 26 tests with 87 assertions takes ~12 seconds.

My questions/concerns

  • I don't mock anything. I feel like I'm doing something wrong because of this but at the same time I am actually providing a "real" environment for the whole application so nothing seems like it needs to be mocked. Is this okay?
  • Am I trying to do too much with each test? I've read articles that warn against having too many dependencies in a test because it makes the test fragile and less effective, but all of the "dependencies" in my tests are actual code -- so if a test breaks it means my actual code is broken which I feel I would want to know.

What else can you tell me? Suggestions? How can I improve this? Thanks!

  • Fantastic performance? My 282 unit tests just passed in 0.275 s. – nhaarman May 7 '15 at 20:18
5

Concept is good enough!

Now, you must decide on what did you build - integration or acceptance tests and support/enhance them for single purpose. Here is the difference:

Integration tests are used by developers to verify that application is basically working and run often from dev machine. TDD tests and bug reproduction tests goes this way. Integration tests should be fast enough to do not bore developer.

Acceptance tests are used mainly by QA to ensure that application build is ready for production. They cover all application functionality including error conditions and tend to run for a while because of large number of tests

Also, you should automate them as most as possible so they can be run in single click - including database setup/restore.

Then you can add them to your Continuous Integration tool so they will run after each commit and have fun!

Regarding your concerns.

Actually concern - that you are testing too much in a single test. Do you have anything complex enough to be tested in isolation? DAL is good candidate since fast tests on DAL can really help when db schema changes. General rule when writing scoped tests is writing them when you believe this code can be broken/will break. Usually that falls to single functions or large isolated module. Tests will large amount of mocks tend to be a) fragile b) expensive to write and c) testing glue code that has no complexity and no value

0

If you like the performance of your tests and you find that they help you catch your mistakes in your production code, that is literally all that's needed (outside of specialized domains where you might have liability or regulatory concerns). If they do these things, great. If they don't, throw them out and try a new methodology.

If you're not sure, I encourage you to "A/B Test" your tests (yo dawg, I heard you like tests...)! Write one your way and write 1+ (probably several) the way you think a unit test is supposed to be. If having lots of collaborators is a problem, that problem will be the difficulty of divining the location of an error in the production code. Try running both tests before you debug either failure and taking a guess based on the error message, then see which one helped you guess closer to the real problem faster in hindsight.

But always: measure the utility tests give you, not their adherence to some standard!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.