Best practices for marking build tarballs as unit/component test passed

Question

During our current build process, we have C code that gets compiled and archived together with some Python code in a tarball. That tarball represents a commit (that can be or not be a release commit).

In some of our Jenkins flows, the code is being tested and in some it isn't. But the information regarding to each specific tarball, whether it passed Unit Test or Component Test remains solely in the Jenkins job, and is gone after the job ends.

Looking at an existing tarball, I can't tell if it passed any of the tests at all, and there's no way I know of, of querying Jenkins about this specific build-flow and the results of its' jobs.

The best options I could think of to mark and save test results are:

1. Add a file to the projects' main directory (the file will be saved inside the tarball) that signifies the tests it passed (e.g. .passedunittest, .passedcomponenttest).
2. Add a text file in the directory where the tarballs are saved, that will act as a database for tarballs, regarding to tests passed or failed.
3. Add a component to the tarball filename that signifies the tests it passed.

I feel fine with all 3 options, but I personally prefer the 1st, but it happens to me that I feel fine with ridiculous solutions. Is there a best practice in this case? And if there is, what is it?

Answer 1

This depends on your workflow. What do you do with the build tarballs?

Adding the test results file to the tarball seems easiest for "manual" / in-place inspection.

OTOH adding a name component allows to use ls, sort, and head to pick the latest successful tarball without inspecting the contents.

If you move the build artifacts around, you can use git-annex to both move them in a controlled manner and associate a commit message (and maybe a signature) with them.

Answer 2

You could use character codes to signal pending, passed or failed builds.

app.SS.tar

The first S means the unit tests were successful. The second S means the component tests were successful.

So successful unit and failing component tests looks like:

app.SF.tar

Successful unit tests and component tests haven't been run yet:

app.SP.tar

Basically each test run gets a character position before the file extension. Then:

• P is Pending
• S is Successful
• F is Failed

Answer 3

Using the commit log to store the test results would be the best practice:

Data are ideal for managing with Git. These include data manually entered via spreadsheets, recorded as part of observational studies, or ones retrieved from sensors (see also section on Managing large data). With each significant change or additions, commits can record a log those activities (e.g. “Entered data collected between 12/10/2012 and 12/20/2012”, or “Updated data from temperature loggers for December 2012”). Over time this process avoids proliferation of files, while the Git history maintains a complete provenance that can be reviewed at any time. When errors are discovered, earlier versions of a file can be reverted without affecting other assets in the project.

References

• Git can facilitate greater reproducibility and increased transparency in science