1

I'm working on a website that, for all intents and purposes, functions as a Choose Your Own Adventure book rendered as a single page app (SPA). Starting at the left, this is the user experience:

enter image description here

Each (virtual) page is rendered from an HTML-like templating language, making the user experience dynamic; for example, the current page's text, choices, and destinations may vary depending on previously viewed text and choices.

Throughout development, a certain class of bugs keep popping up:

  1. A choice should be visible, disabled, or hidden, but it is not.
  2. A destination should have lead to page A, but it went to B instead, or it goes nowhere due to programmer error.
  3. Specific text should exist on the page, but it does not, or vice versa.
  4. A typo in the template language presents a runtime error. Sometimes the error is conditional: 4/5 paths to the page work as expected, but it errors on the last one.

The first three are far more common than the last. Whenever possible, I try to extract all the logic into a function I can test in isolation, but doing so doesn't actually make a significant impact: there will always be the possibility of writing %%if x%% where %%if !x%% was intended. There will always be the possibility of writing %%call y()%% where %%call z()%% was intended.

At the moment, I write automated end to end tests, TDD style to avoid/catch these mistakes. The tests run through scenarios in headless browsers on a local instance of the website.

Here's the problem: they now take ten minutes to complete, killing productivity.

I've considered whether I need such thorough tests. Whether I need so many. But they consistently find bugs, even the ones that check scenarios that haven't changed in months. The main (only?) issue is their speed. But I don't know how to speed them up anymore than I already have, so I'm hoping for some ideas I haven't considered.

How do I improve the speed of the test suite without sacrificing the confidence they afford me?

1
  • There's not really enough context to say. Could you write different kinds of tests? E.g. take a template and a state and render the result for assertions, maybe that would be quicker than getting into that state via the UI.
    – jonrsharpe
    Aug 9, 2023 at 14:54

2 Answers 2

3

The standard solution for dealing with huge test times (or huge build times) is to run the majority of tests on a separate machine, a dedicated test server (or build server, or CI server), automatically after each checkin into your repository.

Of course, since you still want parts of your tests to be executed immediately when doing TDD, this will only work when you split your tests suit into different testing modules which correspond to the modules of your application. So when you work on a module X, run the subset of tests which correspond to module X locally (and maybe some other tests you always want to be executed). The majority of tests, however, should be executed automatically in the background, and when they fail, the test server (or build server, or CI server) sends you a message or email about the failure.

And yes, this will sacrifice "ultra fast feedback" to some degree, but the feedback should be fast enough for most practical purposes. How well this works depends a lot how well you can structure your test suit into parts which can be executed locally and quickly, and other parts where a certain time lag for the feedback is acceptable.

1
  • Great general advice. This applies to any test suite, really. I have a feeling that even after outsourcing test runs, the problem will remain. Sure, 5 minutes (or less) is a heck of a lot better than 10, but I have an inkling that, given the nature of this problem--i.e., it's a web based template language--there's a way to run a huge portion of the suite in under 5 seconds. Maybe I could parse the template files as a string, turn them into an AST, and avoid a browser all together. I don't know; I'm just throwing out ideas, but I'm looking for ideas like that. Aug 9, 2023 at 10:26
3

The heart of your question really is about optimizing code. While there are a few general tricks, your first step is profiling your test code. Only after identifying the steps in each test that perform the worst, can you crunch numbers and determine what is worth optimizing. I'll get to the general advice, but first a short story about optimizing tests, so you don't waste the kind of time that I did.

A Hard Lesson About Optimizing Tests

(which you can skip if you've already learned this lesson)

One of my projects has an extensive test suite using Selenium. I'm not advocating that everyone should have 4,000 tests that open a web browser and perform a use case, but that's what this project has. Our full test suite was taking about 12 hours to run. Suddenly, one day, it was 10:00 AM and the test suite was still running, despite having started the tests at 5:00 PM the previous night.

I spent time looking at how we were initializing our ORM database sessions, connection sharing, how often we open and close the browser window. Nothing seemed to account for the 5+ hours of additional test execution time. The time saved optimizing things that (I guessed) were slow, saved 90 minutes or so. Not bad, but certainly didn't explain why our tests were taking 5 hours more than normal. Until I profiled our tests, and discovered we were traversing a temp directory recursively to delete file uploads before each test.

Deleting these temp files used to take mere milliseconds. Fast-forward 6 months, and I discovered it was taking 7 seconds per test to delete these temp files, because each day we added a new sub folder in the form of year/month/day. The increasing number of sub folders killed performance, even though 98% of our tests did not upload files. Just doing some naïve math, 7 seconds x 4,000 tests is 28,000 seconds — just spent deleting non-existent file uploads! (Imagine me beating my head against a wall at this point) That's nearly 7 hours and 45 minutes of test execution time over the course of the whole test run. And there was my 5+ hours of additional test execution time.

So, I added a step in our build process before executing tests to recursively delete all files and folders in this temp folder (akin to rm -rf /uploads/temp). Sure 98% of our tests didn't upload files, so we don't need to delete temp files before every test, but now at least deleting non-existent files takes mere milliseconds again.

So, lesson learned: always, always, always, always, always, always, always, always, always, always, always, always, always, always, always, always profile your application or tests before wasting time optimizing things. And why this is a lesson I continually need to learn is another question indeed.

Ok, so enough anecdotes. Let's assume you have profiled your tests and eliminated the low hanging fruit, performance-wise. You still have 1,000+ tests to execute, and they simply take time. Too much time for a quick feedback loop during development.

General Advice For Optimizing Slow UI Tests

  • Execute tests in parallel. This isn't always possible and depends on the setup for your tests and application environment. I've done this before. Now instead of 1,000 tests running, you need to wait for the equivalent of 500 tests to run since maybe you can run 2 simultaneously.

  • Install an SSD drive. Yup. The old "throw more hardware at the problem" trick. When tests become I/O bound, make I/O faster.

    • Or a faster CPU or more RAM.
  • Don't rely on slow UI tests during development. You need tests that execute faster, and the UI, along with a server and database, can only execute so fast. This requires structuring your code for unit tests so you don't need a web server or database. You can mock any calls to external resources. This is pretty tricky for UI components, but not impossible. It is a lot of work, though, so the increase in test speed should be enough to offset the additional work to write unit tests for UI code.

Unfortunately the only true solution to your problem is "Don't rely on slow UI tests during development." You will need to structure your UI components so they can be isolated from the rest of the page, and then write unit tests for them. They still might execute asynchronously, but they shouldn't be relying on real servers or databases. All calls should be mocked so the tests execute quicker.

Otherwise, you are stuck with slow UI tests during development.

4
  • That's good advice. I'm using testcafe and followed all the advice in this article. That's probably why this suite takes 10 minutes instead of 20. Beyond that, I don't know how to do any more due diligence; I can run the browser in normal mode (as opposed to headless mode) and observe with my eyes that there's seconds of delay between the browser opening and the test starting, but I can't reproduce this outside of that situation. I don't even know if it's relevant since I run the tests in headless mode? Aug 17, 2023 at 2:48
  • That's one of many examples that makes it hard to pinpoint exactly where the performance issue lies. I assume that delay is the time it takes for testcafe to set up its environment, but can't prove it. I thought rewriting a simple test to run without a browser (i.e., using testing-library or jsdom) would be the pragmatic next step. By doing that, I could theoretically rule out a bunch of technologies simultaneously. After struggling for days, I started wondering, maybe there's a best practice for testing browser templating languages, hence the original question. Aug 17, 2023 at 3:04
  • @DanielKaplan: you don't test the template language, you test the behavior. And for that you either need objects or functions to interact with so you can write unit tests, or realize that testing the UI layer of an application just takes a long time. As for profiling your tests, it can be simple log statements that subtract the time between two events. I'm just not sure how to log to the test output. Aug 17, 2023 at 12:34
  • "you don't test the template language, you test the behavior" Agreed. Isn't the bullet list in my original question a list of behaviors I want to test? The paragraph that follows is my attempt to explain why interacting with objects and functions doesn't fully test those behaviors. I guess that leaves me with one option. Aug 17, 2023 at 14:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.