233 votes
Accepted

What's the point of running unit tests on a CI server?

Surely, by the time something gets committed to master, a developer has already run all the unit tests before and fixed any errors that might've occurred with their new code. Or not. There can be ...
Oded's user avatar
  • 53.5k
156 votes

If two individual branches pass unit tests, once they're merged, is the result also guaranteed to pass unit tests?

No. The simplest example I've seen is: branch A cleans unused imports in a file. Branch B adds code that actually uses some of the unused imports. Git merges automatically since the lines that were ...
Esben Skov Pedersen's user avatar
114 votes

If two individual branches pass unit tests, once they're merged, is the result also guaranteed to pass unit tests?

No. As a counter example, consider branch A adds a unit test that uses reflection to check for a misspelling in an enum. And branch B adds a misspelling. Both pass because a misspelling doesn’t fail a ...
Telastyn's user avatar
  • 109k
78 votes

Is the usage of random values in unit testing a good practice?

Yes, I agree that randomness shouldn't be part of a testing suite. What you want is to mock any real randomness, to create deterministic tests. Even if you genuinely need bulk random data, more than ...
Flater's user avatar
  • 47.7k
77 votes

What's the point of running unit tests on a CI server?

As a developer who doesn't run all the integration and unit tests before making a commit to source control, I'll offer up my defense here. I would have to build, test and verify that an application ...
TZHX's user avatar
  • 5,062
68 votes
Accepted

How do we avoid CI-driven development...?

CI-driven development is fine! This is a lot better than not running tests and including broken code! However, there are a couple of things to make this easier on everyone involved: Set expectations: ...
amon's user avatar
  • 134k
54 votes

TODO comments with deadlines?

This question is really two questions in one. Todo comments Of all the ways to track action items, this is the worst. TODO comments are good during active work or as a way of suggestion to a ...
Brandon's user avatar
  • 4,565
52 votes

Is the usage of random values in unit testing a good practice?

I've worked on projects which use anywhere from no to extensive randomness in tests, and I'm generally in favour of it. The most important thing to remember is that the randomness must be repeatable. ...
l0b0's user avatar
  • 11.3k
47 votes

Unexpected Code Coverage Reduction

The problem I see here is that you have made the code coverage a trigger for build failure. I do believe that code coverage should be something that is routinely reviewed, but as you have experienced,...
Berin Loritsch's user avatar
37 votes

Unexpected Code Coverage Reduction

Have you considered not using code coverage metrics? I'm not going to argue that code coverage isn't something that you should look at. It absolutely is. It's good to keep track of what was covered ...
Thomas Owens's user avatar
  • 80.4k
36 votes

If two individual branches pass unit tests, once they're merged, is the result also guaranteed to pass unit tests?

Here is an example which neither does require changes to the existing tests itself, nor reflection, nor a failing build, for not giving the wrong impression such cases can only happen under artificial ...
Doc Brown's user avatar
  • 203k
34 votes

How do we avoid CI-driven development...?

Building a sustainable plugin model requires that your core framework expose a stable interface that plugins can rely on. The golden rule is that you can introduce new interfaces over time but you can ...
casablanca's user avatar
  • 4,934
32 votes
Accepted

How can CI be used for interpreted languages?

Continuous integration as a term refers to two distinct ideas. The first is a workflow: instead of everyone in a team working on their own branch and then after a couple of weeks of programming try ...
amon's user avatar
  • 134k
27 votes

Which comes first: CD/Trunk-based development or microservices?

Consensus is you want CI/CD first and anyway, independent of your application's language, design or architecture. Whether you deliver trunk-based or use feature branching is also independent of CI/CD. ...
Martin Maat's user avatar
  • 18.3k
24 votes

How can CI be used for interpreted languages?

True, you do not have particular need of a CI system to perform builds and check that those builds are correct, but that is only part of what CI is about. The purpose of CI is to detect errors as ...
Iker's user avatar
  • 865
24 votes

TODO comments with deadlines?

Do not use TODOs. You already have a TODO list in your project. It's called the issue tracker. I think the real problem is in this sentence: we can create a ticket in our issue management system, ...
Jens Schauder's user avatar
24 votes
Accepted

continuous integration for scientific software

Testing scientific software is difficult, both because of the complex subject matter and because of typical scientific development processes (aka. hack it until it works, which doesn't usually result ...
amon's user avatar
  • 134k
23 votes

What's the point of running unit tests on a CI server?

Apart from the excellent Oded answer: You test the code from the repository. It may work on your machine with your files... that you forgot to commit. It may depend on a new table that does not have ...
Borjab's user avatar
  • 1,339
23 votes

What's the point of running unit tests on a CI server?

You'd think so wouldn't you - but developers are human and they sometimes forget. Also, developers often fail to pull the latest code. Their latest tests might run fine then at the point of check-in, ...
Robbie Dee's user avatar
  • 9,737
22 votes
Accepted

Keep branches from piling up

It sounds like you have a few problems here: 1. Identifying features for a specific release This is a project management issue, and a coordination issue. Will this feature be released before, at the ...
Jen's user avatar
  • 693
18 votes
Accepted

Unexpected Code Coverage Reduction

You can mitigate the effect to some degree by allowing the relative code coverage to reduce when the total number of uncovered lines also reduces, or when the total number of lines reduces, since this ...
Doc Brown's user avatar
  • 203k
17 votes

What does "continuous" mean in "continuous deployment", "continuous delivery" and "continuous integration"?

Traditionally, before Continuous Integration, Continuous Build, Continuous Deployment, Continuous Testing, etc., the system was split into modules and those modules were developed by independent teams ...
Jörg W Mittag's user avatar
16 votes

When do you have enough automatic testing to be confident in your continuous integration pipeline?

In General When do you have enough automatic testing to be confident in your continuous integration pipeline? The answer probably becomes clear if you think about what you want to be confident ...
R. Schmitz's user avatar
  • 2,608
15 votes

Unexpected Code Coverage Reduction

This is called Simpson’s paradox, and is a well known statistical issue with your approach. You could even construct cases where you refactor and afterwards every single method has a higher coverage, ...
Aganju's user avatar
  • 1,463
14 votes

What's the point of running unit tests on a CI server?

by the time something gets committed to master I usually set up my CI to run on every single commit. Branches don't get merged into master until the branch has been tested. If you're relying on ...
Daenyth's user avatar
  • 8,097
12 votes

When to do code reviews when doing continuous integration?

Here is the extract from continuous delivery author: Jez Humble Writes as: I am currently writing a blog post on this topic. The short answer is this: The best way to review code is through pair ...
Alagesan Palani's user avatar
12 votes
Accepted

Where to push a failing test?

What I'd do in this situation is to mark the failing tests as "ignored" - that way you still have the test so that you know what you need to fix in future, but you're not going to end up with broken ...
Philip Kendall's user avatar
12 votes
Accepted

when should unit tests be done in a continuous integration environment

Usually the developer runs them manually before checking in, and a CI server runs them automatically after checking in. Programmers are usually pretty good about running unit tests for incremental ...
Karl Bielefeldt's user avatar
12 votes
Accepted

Should testers have root access or not, on personal VM, given that developers do?

The situation looks pretty simple to me: people need the tools to do their job correctly. If they need root access, they should be given root access and they should be trusted (and trained, if ...
Bryan Oakley's user avatar
  • 25.3k
12 votes
Accepted

How is continuous integration organized in large companies?

It is, basically, a scaling problem. You separate your work into modules, which can be different projects and/or different functionalities of your product. You would have teams that cover sets of ...
Vladimir Stokic's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible