How to make the tests run faster?

Question

I develop my current project with a very strict approach to TDD and try to make it as close to Kent Beck's TDD By Example book. So right now I've got code coverage of 93%, with 1290 unit test cases, 58 integration tests, and 25 UI tests. However, the tests durations on the CI are too long IMHO:

• Unit tests: 15 minutes
• Integration tests: 10 minutes
• UI tests: 30 minutes

And this will add up when I need to checkout and compile the dependencies from the Carthage. So much so that it was canceled by the CI because it took too long. Can you suggest me a good way to approach the testing?

A few facts and questions about the project:

1. I separate the tests by the effects of the method. For example, consider a method called loadData(), this will have these test cases:

   • testLoadData_ShouldEmitDataFromApi()
   • testLoadData_ShouldDisableUI_Initially()
   • testLoadData_ShouldShowLoadingIndicator_Initially()
   • testLoadData_WhenApiSuccessful_ShouldEnableUI()
   • testLoadData_WhenApiSuccessful_ShouldHideLoadingIndicator()
   • testLoadData_WhenApiFailed_ShouldShowErrorMessage()
   • testLoadData_WhenApiFailed_ShouldEnableUI()
   • testLoadData_WhenApiFailed_ShouldHideLoadingIndicator()

   Is this the right approach? Or can it be simplified more?

2. I've set the timeout for integration and UI tests to 5 seconds. At first, I use 2 seconds but it returned lots of false negatives because of timeouts on the CI. These tests got red almost 20-30% of the times with 2 seconds timeout. From your experience, what is the proper timeout that you usually use?

3. I've got 4 main flows in the CI:

   • Development: triggered when there's a push to any branch besides master
   • Pull Req: triggered when there's a pull req update
   • Master: triggered when there's a merge to master (ie. when a PR is merged)
   • Store: triggered manually when we want to build for the store

   All of these flows always run all the tests (ie. unit, integration, UI). Should I disable some of the tests on a few of the flows? I was thinking that the Development flow only need to test for unit tests, and the rest can be run in Pull Req flow, while Store will only need to test for UI tests? Or something else.

Thanks.

EDIT: I forgot to point out that the timing was from the CI. If I do the Unit tests on my own local machine, it will just be done in 2-3 minutes tops. And the whole tests will be done in like 15-20 minutes.

But after reading some of the comments and answers here, I decided to take a look at the CI reports again because I find them odd. And I find out that the one that takes a long time was actually the building of the project since it needs to also build the dependencies from Cocoapod and SPM. They always get built because I always delete the DerivedData folder that has the cached build of those dependencies at the start. The CI reported both the building process and the testing as the same entry, xcode-tests. And the timing is actually the combined timing of the building and testing of the project. So I think I've found the major cause of this problem.

Regarding the unit tests, I think I've set them properly. I isolated them into their own units and I use mock classes or functions and injected them as the dependencies. For instance, the test cases above will be like so:

class DashboardViewModelTest: XCTestCase {
    private let disposeBag: DisposeBag!

    override func setUpWithError() throws {
        try super.setUpWithError()
        disposeBag = DisposeBag()
    }

    func testLoadData_ShouldEmitDataFromApi() {
        let expectedUserId = "user1234"
        let expectedEmittedContents = [Content.random(), Content.random(), Content.random()]
        var actualUserId: String?
        let dataProvider: (String) -> Observable<[Content]> = {
            actualUserId = $0
            return .just(expectedEmittedContents)
        }
        let viewModel = DashboardViewModel(dataProvider: dataProvider)
        
        var actualEmittedContents: [Content]?
        viewModel.dataLoaded
            .emit(onNext: { actualEmittedContents = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: expectedUserId)

        XCTAssertEqual(expectedUserId, actualUserId)
        XCTAssertEqual(expectedEmittedContents, actualEmittedContents)
    }

    func testLoadData_ShouldDisableUI_Initially() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .never() })
        
        var isUiEnabled: Bool?
        viewModel.uiEnabled
            .drive(onNext: { isUiEnabled = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertFalse(isUiEnabled)
    }
   
    func testLoadData_ShouldShowLoadingIndicator_Initially() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .never() })
        
        var isLoadingIndicatorShown: Bool?
        viewModel.loadingIndicatorShown
            .drive(onNext: { isLoadingIndicatorShown = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertTrue(isLoadingIndicatorShown)
    }

    func testLoadData_WhenApiSuccessful_ShouldEnableUI() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .just([Content.random()]) })
        
        var isUiEnabled: Bool?
        viewModel.uiEnabled
            .drive(onNext: { isUiEnabled = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertTrue(isUiEnabled)
    }

    func testLoadData_WhenApiSuccessful_ShouldHideLoadingIndicator() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .never() })
        
        var isLoadingIndicatorShown: Bool?
        viewModel.loadingIndicatorShown
            .drive(onNext: { isLoadingIndicatorShown = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertFalse(isLoadingIndicatorShown)
    }

    func testLoadData_WhenApiFailed_ShouldShowErrorMessage() {
        let expectedErrorMessage = "Random error happened!"
        let errorWithMessage = ErrorWithMessage(message: expectedErrorMessage)
        let viewModel = DashboardViewModel(dataProvider: { _ in .error(errorWithMessage) })
        
        var actualErrorMessage: String?
        viewModel.showError
            .emit(onNext: { actualErrorMessage = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertEqual(expectedErrorMessage, actualErrorMessage)
    }

    func testLoadData_WhenApiFailed_ShouldEnableUI() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .error(Error()) })
        
        var isUiEnabled: Bool?
        viewModel.uiEnabled
            .drive(onNext: { isUiEnabled = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertTrue(isUiEnabled)
    }

    func testLoadData_WhenApiFailed_ShouldHideLoadingIndicator() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .error(Error()) })
        
        var isLoadingIndicatorShown: Bool?
        viewModel.loadingIndicatorShown
            .drive(onNext: { isLoadingIndicatorShown = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertFalse(isLoadingIndicatorShown)
    }
}

Answer 1

Your tests are doing way too much.

Your unit tests take on average nearly a second. A typical unit test's running time is less than 10 milliseconds.

It's more common for integration tests to take longer, but 10 minutes is still a lot. UI tests being very slow is normal (it's very common to run those only on certain parts of the CI flow, like nightly).

You're not giving any information about how your tests are set up and run, but my guess is that your "unit tests" are not actually unit tests, that you're not testing the code in isolation and they access the filesystem or a DB, or involve a dependency injection container that has to read config files and scan the entire source code.

A proper unit test runs only the code being tested (which may include multiple classes depending on your interpretation of "unit"), but not frameworks, libraries or other dependencies.

Of course, it's also possible that there's just some silly configuration error that somehow results in some costly but unnecessary setup for every test. (Edit: looks like this was in fact the case)

Using a profiler may give you more insights into what is actually taking so long.