Projects like webkit, Linux kernel and many others take over ten minutes if not hours for a full build. What's the practical development cycle for these large projects? Specifically, how does a developer fix and test an identified bug? Can you give some concrete examples?
4
-
Well, our CI server agents are pretty fast :3. Even with the larger system parts. Build and Unit-Tests of new code submitted to the repository, is triggered immediately for any commit to one of the relevant repositories. Takes from 6-8 minutes to respond the dev(s) in charge. Sys-Integration tests run after full build of the whole deployment package, publishing to the QA is done with manually issued pins.– πάντα ῥεῖJul 1, 2018 at 9:55
-
9Incremental builds, partial builds, etc..– EuphoricJul 1, 2018 at 10:08
-
Check a professional CI build system like Team City. There are all kinds of features to optimize your builds in regards to changes committed to you underlying repositories.– πάντα ῥεῖJul 1, 2018 at 11:22
-
The Linux kernel (25MLOC only) can take less than half an hour of build time on a high-end desktop. This is not prohibitive, and incremental builds are shortening that. But large proprietary products (think of google indexing engine, rumored to be at least 500MLOC linked in a single executable) are a different story– Basile StarynkevitchJul 1, 2018 at 13:28
Add a comment
|
4 Answers
Projects like webkit, Linux kernel and many others take over ten minutes if not hours for a full build.
Ten minutes or half an hour is not prohibitive build time.
When I started my career (1987, Sun3/160 workstation on Unix) the full (cold) build of a 200KLOC program that I wrote alone took more than half an hour.
With incremental builds (combined with parallel builds, perhaps as crude as make -j, or even parallel and distributed builds like with distcc, icecream, or even better ) and a fast (multi-core) desktop, a Linux kernel (only 25MLOC) builds in typically less than ten minutes (and most incremental builds need less than a minute; so rebooting the new kernel could take more time than rebuilding it). That is ok. See this benchmark (it shows that a Linux kernel can be cold built in less than 15 minutes on a powerful desktop)
And in the 1970s, programmers (then coding programs less than 2KLOC on punched cards) could often afford only one or perhaps two or three compilations per day. They just thought more carefully of their own code. Their attitude was different (not thinking something like: I forgot the syntax for that, just will compile it).
Sorry to sound condescending (but I was born in 1959), but you are a spoiled kid if you cannot wait ten minutes (or even half an hour) for a build. But I am spoiled too these days....
Of course, you should think about your software build and design it wisely.
Some proprietary products (think of Google indexing engine) are rumored to have more than half a billions lines of C++ code (which build much slower than C, into a single executable). The link time is then an issue. But Google funded Ian Lance Taylor to design and develop gold (precisely to accelerate that). And Google developed their own build automation engines... and uses high-end servers for that.
Some current HPC code (e.g. simulation of galaxy collision) needs today weeks of CPU time. Debugging them is probably harder, if the bug needs a whole day to occur (and is a numerical bug). But then the code is designed differently (e.g. you think of persistence at code design time).
Sometimes, a reproducible bug may take 30 minutes of CPU time to be reached (and that could happen even in a tiny personal project of only two thousands lines), and you have to live with that (if you have designed your program with some persistency in mind, things are different; however adding persistence or checkpointing to some existing program is generally so large an architectural change that you can't afford it).
Also, modularity is now widely used (even C++20 might have modules). Most programming environments these days have some notion of libraries. Many software projects use the PIMPL idiom. And recent programming languages (Go, Rust, Ocaml) know about modules. Also, huge software systems tend to be distributed and split in several executables (e.g. microservice approach), in particular because you want to take advantage of process isolation (which requires to define more carefully program interfaces).
At last, the total volume of software is growing much less than the data managed by it (look into sofwareheritage for a clue; an estimate of the total source code size on Earth is perhaps a few hundreds of terabytes, many orders of magnitudes less than the total amount of data on the Internet = many exabytes, perhaps a few zettabytes). Even an entire Linux distribution (about 20GLOC of code) can be built in less than a week on a desktop. Try some source code Linux distribution (e.g. gentoo).
You might look inside several large free software projects (e.g. Qt, GTK, GCC, Libreoffice, Firefox, Linux or *BSD kernels, Java JRE, ...) to understand more how they are built and developed by their open communities. Details would vary (because the architecture and design of these software, the programming language and build automation tools is different).
Notice that in 2018 and in most developed countries, the cost of skilled labour (those of competent developers) is much more than the cost of computers to build software. So it makes sense to provide developers with powerful build systems.
answered Jul 1, 2018 at 14:15
-
4"Sorry to sound condescending (but I was born in 1959), but you are a spoiled kid if you cannot wait ten minutes (or even half an hour) for a build." TIL I'm incredibly spoiled.– TagcJul 2, 2018 at 7:46
-
Because you can afford it (and so do I today). In some situations, you have to wait 20 minutes for a build to complete, and you live with that. In more situations, you have to run your buggy program for 30 minutes in your debugger to spot and reach a crashing bug Jul 2, 2018 at 7:53
-
1"Sorry to sound condescending (but I was born in 1959), but you are a spoiled kid if you cannot wait ten minutes (or even half an hour) for a build." The issue here isn't so much personal impatience. You need to consider the drastically changed development environment. There are plenty of employers who track developer work time (e.g. Jira), where an extra 30min is considered reason for inquiring why a minor bug took so long. I do agree that that's a mistake made by management (as they should factor build time into it), but OP's question is not necessarily indicative of OP being spoilt.– FlaterAug 23, 2018 at 9:32
-
Those employers should be aware of the long build time. Then decide (or not) to reduce it Aug 23, 2018 at 10:25
If a full build takes prohibitively long, so that you cannot test the system after every change, there are three things you can do:
- make the full build faster
- don't make a full build
- don't test
Real-world projects do any one of (or any combination of) those three.
answered Jul 1, 2018 at 11:14
-
-
3@πάνταῥεῖ, Let's say your code runs on dozens of different processors and other factors so there is a huge matrix of configurations to test -- so, maybe skip testing some configurations sometimes, in between releases, using appropriate judgement. Jul 1, 2018 at 14:58
-
@πάνταῥεῖ: The problem has three components: a) it takes too long b) to do a full build c) to test every bugfix. You can solve the problem by changing any one of the three components: a) make the build faster, b) don't make a full build (e.g. partial or incremental), c) don't test every change (e.g. only test over night, batching multiple changes). Jul 1, 2018 at 21:51
-
6@πάνταῥεῖ: One project I know of has a 9 hour build phase that cannot be shortened. (It is a hardware-software co-design, and synthesizing the FPGA takes 9 hours on the fastest machine that the vendor's proprietary toolchain will run on-) So, the guys can either a) wait 9 hours after every little change, or b) not test every little change. Instead, they fix multiple bugs, develop multiple features and make multiple changes during the day, run the build over night, and test all changes of the previous day combined in the morning. Jul 1, 2018 at 21:51
What's the practical development cycle for these large projects?
- Allow developers to commit their code1
- Install triggers with your preferred CI System (Jenkins, Team City, etc.)
- Give devs as fast as possible response, regarding any build or unit test fails
- After all low level build artifacts collect them to provide a deployable build and start system integration tests on them
- Let some person in charge decide, if the build met all requirements for a QA relevant testing milestone and publish it
Specifically, how does a developer fix and test an identified bug?
They should get some short time response from the CI system as mentioned.
Of course their will be all kind of compiler error and unit test case results pointing them where exactly to dig in with fixing their dev environment, or starting the debugger and try the unit tests getting green again.
Can you give some concrete examples?
Of course I could, but that seems to be a bit futile here, since all those examples how to fix such bug may vary in a very broad manner.
Reasons can vary from false positive fails detected by the build server to blatantly hardcoded values in the source code.
The main point in managing a larger group of developers in a large scale project is you should notify them about such failures as fast as possible.
1)Though make them aware, that they shouldn't leave their workplace being unavailable until the CI system signals them green.
-
3Our industry needs to lose this notion that if a build breaks an engineer has to be on call 24/7 to fix it. Just back out the change, notify the committer, and move on. If I work 9 to 5 and my build takes 6 hours, it's stupid that any time after 11AM, I'm expected to stick around for 6 hours to make sure nothing went wrong. That's just not an acceptable way to treat people. Jul 2, 2018 at 6:14
-
@user1118321 Well, I probably forgot to mention that our source code + unit test builds usually complete within 6-8 minutes. Jul 2, 2018 at 6:16
-
@user1118321 Using a gating CI system (based on pre-commit verifications) can eliminate such frustration - devs don't commit directly, instead they submit candidate changes to the CI system which automatically commits them after actually verifying that they pass the quality checks. The faulty changes are simply rejected/returned to the submitter for reconsideration without causing blockages at the project level - no more blame and need for immediate action. Jul 5, 2018 at 11:14
-
The requirement of a fast feedback from the CI system for each and every changeset is also much more relaxed, the process can easily scale to massive projects with significant verification times and 100s/1000s of devs (depending on the CI system's orchestration algorithm, of course). Jul 5, 2018 at 11:19
A very effective approach is using a gating CI system capable of bundling multiple candidate changesets for pre-commit verifications.
Developers can use in their own environment/workspace incremental builds to speedup preparing their changesets. But they don't commit them directly to the integration branch - their private verifications aren't reliable enough. Instead they submit them to the gating CI system for the final verification and (automated!) commit/merge into the integration branch.
As (positive IMHO) side effects such process can:
- guarantee prevention of breakages, since the centralized, orchestrated pre-commit CI verifications can be made 100% reliable
- eliminate blame - the CI system is responsible for any breakages, if any, not the developers
One such CI system I designed for a previous employer was decently serving over 1000 devs doing trunk based development on the main branch, with commit gating criteria including 4-6h builds and 5-12h smoke tests, at an avg 60-80 commits/day rate.
Another example would be OpenStack's CI gating system based on Zuul/Gerrit. A couple of presentations about it:
