0

I develop my current project with a very strict approach to TDD and try to make it as close to Kent Beck's TDD By Example book. So right now I've got code coverage of 93%, with 1290 unit test cases, 58 integration tests, and 25 UI tests. However, the tests durations on the CI are too long IMHO:

  • Unit tests: 15 minutes
  • Integration tests: 10 minutes
  • UI tests: 30 minutes

And this will add up when I need to checkout and compile the dependencies from the Carthage. So much so that it was canceled by the CI because it took too long. Can you suggest me a good way to approach the testing?

A few facts and questions about the project:

  1. I separate the tests by the effects of the method. For example, consider a method called loadData(), this will have these test cases:

    • testLoadData_ShouldEmitDataFromApi()
    • testLoadData_ShouldDisableUI_Initially()
    • testLoadData_ShouldShowLoadingIndicator_Initially()
    • testLoadData_WhenApiSuccessful_ShouldEnableUI()
    • testLoadData_WhenApiSuccessful_ShouldHideLoadingIndicator()
    • testLoadData_WhenApiFailed_ShouldShowErrorMessage()
    • testLoadData_WhenApiFailed_ShouldEnableUI()
    • testLoadData_WhenApiFailed_ShouldHideLoadingIndicator()

    Is this the right approach? Or can it be simplified more?

  2. I've set the timeout for integration and UI tests to 5 seconds. At first, I use 2 seconds but it returned lots of false negatives because of timeouts on the CI. These tests got red almost 20-30% of the times with 2 seconds timeout. From your experience, what is the proper timeout that you usually use?

  3. I've got 4 main flows in the CI:

    • Development: triggered when there's a push to any branch besides master
    • Pull Req: triggered when there's a pull req update
    • Master: triggered when there's a merge to master (ie. when a PR is merged)
    • Store: triggered manually when we want to build for the store

    All of these flows always run all the tests (ie. unit, integration, UI). Should I disable some of the tests on a few of the flows? I was thinking that the Development flow only need to test for unit tests, and the rest can be run in Pull Req flow, while Store will only need to test for UI tests? Or something else.

Thanks.

EDIT: I forgot to point out that the timing was from the CI. If I do the Unit tests on my own local machine, it will just be done in 2-3 minutes tops. And the whole tests will be done in like 15-20 minutes.

But after reading some of the comments and answers here, I decided to take a look at the CI reports again because I find them odd. And I find out that the one that takes a long time was actually the building of the project since it needs to also build the dependencies from Cocoapod and SPM. They always get built because I always delete the DerivedData folder that has the cached build of those dependencies at the start. The CI reported both the building process and the testing as the same entry, xcode-tests. And the timing is actually the combined timing of the building and testing of the project. So I think I've found the major cause of this problem.

Regarding the unit tests, I think I've set them properly. I isolated them into their own units and I use mock classes or functions and injected them as the dependencies. For instance, the test cases above will be like so:

class DashboardViewModelTest: XCTestCase {
    private let disposeBag: DisposeBag!

    override func setUpWithError() throws {
        try super.setUpWithError()
        disposeBag = DisposeBag()
    }

    func testLoadData_ShouldEmitDataFromApi() {
        let expectedUserId = "user1234"
        let expectedEmittedContents = [Content.random(), Content.random(), Content.random()]
        var actualUserId: String?
        let dataProvider: (String) -> Observable<[Content]> = {
            actualUserId = $0
            return .just(expectedEmittedContents)
        }
        let viewModel = DashboardViewModel(dataProvider: dataProvider)
        
        var actualEmittedContents: [Content]?
        viewModel.dataLoaded
            .emit(onNext: { actualEmittedContents = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: expectedUserId)

        XCTAssertEqual(expectedUserId, actualUserId)
        XCTAssertEqual(expectedEmittedContents, actualEmittedContents)
    }

    func testLoadData_ShouldDisableUI_Initially() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .never() })
        
        var isUiEnabled: Bool?
        viewModel.uiEnabled
            .drive(onNext: { isUiEnabled = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertFalse(isUiEnabled)
    }
   
    func testLoadData_ShouldShowLoadingIndicator_Initially() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .never() })
        
        var isLoadingIndicatorShown: Bool?
        viewModel.loadingIndicatorShown
            .drive(onNext: { isLoadingIndicatorShown = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertTrue(isLoadingIndicatorShown)
    }

    func testLoadData_WhenApiSuccessful_ShouldEnableUI() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .just([Content.random()]) })
        
        var isUiEnabled: Bool?
        viewModel.uiEnabled
            .drive(onNext: { isUiEnabled = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertTrue(isUiEnabled)
    }

    func testLoadData_WhenApiSuccessful_ShouldHideLoadingIndicator() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .never() })
        
        var isLoadingIndicatorShown: Bool?
        viewModel.loadingIndicatorShown
            .drive(onNext: { isLoadingIndicatorShown = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertFalse(isLoadingIndicatorShown)
    }

    func testLoadData_WhenApiFailed_ShouldShowErrorMessage() {
        let expectedErrorMessage = "Random error happened!"
        let errorWithMessage = ErrorWithMessage(message: expectedErrorMessage)
        let viewModel = DashboardViewModel(dataProvider: { _ in .error(errorWithMessage) })
        
        var actualErrorMessage: String?
        viewModel.showError
            .emit(onNext: { actualErrorMessage = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertEqual(expectedErrorMessage, actualErrorMessage)
    }

    func testLoadData_WhenApiFailed_ShouldEnableUI() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .error(Error()) })
        
        var isUiEnabled: Bool?
        viewModel.uiEnabled
            .drive(onNext: { isUiEnabled = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertTrue(isUiEnabled)
    }

    func testLoadData_WhenApiFailed_ShouldHideLoadingIndicator() {
        let viewModel = DashboardViewModel(dataProvider: { _ in .error(Error()) })
        
        var isLoadingIndicatorShown: Bool?
        viewModel.loadingIndicatorShown
            .drive(onNext: { isLoadingIndicatorShown = $0 })
            .disposed(by: disposeBag)

        viewModel.loadData(id: "random")

        XCTAssertFalse(isLoadingIndicatorShown)
    }
}
5
  • 4
    Going by your numbers, your unittests take on average about 500ms per test to complete. I am used to single-digit execution times with sometimes an outlier going into tens of ms, so what is taking your unit tests so long? May 21 at 6:16
  • 1
    Remember that it's good to have separate tests for testing separate effects, but that doesn't mean you have to repeat fixtures unnecessarily. If a loading method has four effects to test, by all means write four tests, but they can simply test the same fixture. This doesn't compromise test independence at all. May 21 at 6:33
  • I think I've found the cause. The details are in the edit above. May 21 at 8:33
  • Why is 15 minutes too long on a CI server? It's not like you're waiting for it to complete before you start coding again, are you? May 21 at 15:35
  • @RobertHarvey It's because it will be around 40-45-ish with the whole tests (unit, integration, UI). Plus if I have an update on the Carthage dependencies, the whole building of the Carthage is around 35. Not to mention if it's on the Store flow where I need to archive and deploy to App Store. It will be over 90 minutes for the whole flow. And unfortunately, there's a limit of 90 minutes on the CI before the process got canceled by the CI itself. Furthermore, the cost of the CI is proportional with the duration. So, the faster the better. May 22 at 23:31
8

Your tests are doing way too much.

Your unit tests take on average nearly a second. A typical unit test's running time is less than 10 milliseconds.

It's more common for integration tests to take longer, but 10 minutes is still a lot. UI tests being very slow is normal (it's very common to run those only on certain parts of the CI flow, like nightly).

You're not giving any information about how your tests are set up and run, but my guess is that your "unit tests" are not actually unit tests, that you're not testing the code in isolation and they access the filesystem or a DB, or involve a dependency injection container that has to read config files and scan the entire source code.

A proper unit test runs only the code being tested (which may include multiple classes depending on your interpretation of "unit"), but not frameworks, libraries or other dependencies.

Of course, it's also possible that there's just some silly configuration error that somehow results in some costly but unnecessary setup for every test. (Edit: looks like this was in fact the case)

Using a profiler may give you more insights into what is actually taking so long.

1
  • I've added some more information about this. And I think I've also found the cause. However, I think I will reduce the number of integration and UI tests to only when the PR is being merged to master. Like you suggested. Thanks. May 21 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.