23

I have a C header that is generated from a CSV file and a python script. The C header mainly contains a list of #define constants.

I want to be able to detect manual changes to this header during compilation (which tends to happen frequently in this early phase of development), and have the compiler display a warning to indicate to the developer to update the CSV file and regenerate the header.

If I were to go about doing this, I would have the python script generate some kind of metadata about the file itself, perhaps a hash, and then the compiler would somehow check this hash and compare to what's in the file. But I'm not sure what's the best way to go about it. Does GCC have any facilities I can use for this kind of thing?

  • 36
    Sounds like an X-Y-problem. – Doc Brown Feb 16 at 8:17
  • How long does it takes to build your entire program (hours, or minutes)? On what operating system? – Basile Starynkevitch Feb 16 at 8:45
  • What kind of program are you writing? Some bootstrapped compiler? Without more details and motivation, your question remains unclear. I vote to close it – Basile Starynkevitch Feb 16 at 8:48
  • 1
    have you thought about just checking the timestamp in your build script? if header is newer then CSV file, reject? – Polygnome Feb 16 at 19:19
  • 10
    make is designed for this. – Thorbjørn Ravn Andersen Feb 17 at 16:19
7

Does GCC have any facilities I can use for this kind of thing?

Not that I am aware of such feature.

But you could do something like generating a MD5 checksum after you generated the header file and put that into a different file (e.g. header_name.md5).
Then you could setup a pre-build step to check these by comparison in your build system.


As for your comment that you'd rather like to keep the hash in the file itself:

That's certainly doable but will complicate things a bit in the following ways:

  1. You need to rebuild the hash in the pre-build step without the number kept in a special comment tag or such (e.g. filter it out and use some standard tool to build the MD5 hash).
  2. Building a MD5 hash simply from a file and store it elsewhere is fairly easy. Putting it into the original file itself is an extra step.

I'd not be worried that much about "cluttering" the source tree with extra files. If these have their meaning and importance in the overall build process, I'd prefer to keep the pre-build step simple and concise for anyone.


Moreover despite it doesn't give an answer to your actual problem you're trying to solve, I very much agree with @docbrown's answer here.
Just make it clear from documenting that those generated files shouldn't be changed manually.

  • I've been trying to avoid cluttering with extra files and would much rather embed this info in the file itself – 9a3eedi Feb 16 at 7:46
  • 2
    @9a3eedi You could also put that in the header file itself, but that would make it harder to implement the pre-build step because you'll need some extra parsing to extract the MD5 hash. But that's doable of course (e.g. use a special comment tag to indicate the checksum). – πάντα ῥεῖ Feb 16 at 7:49
  • 1
    @9a3eedi I've done the pattern where you put it in a comment and then strip it out when calculating the checksum. It works, but does call for a lot of specialized build steps that have to be maintained. Another approach to minimize the clutter of files is to put all of the checksums in one file – Cort Ammon Feb 17 at 5:35
  • The MD5 approach will just make people update the MD5 file as part of their workflow, then they'll complain about the workflow being more complex than necessary. I do not recommend it. – user253751 Feb 17 at 10:32
  • I have marked this as the answer as it's the closest to answering the question. However the other answers are excellent and I will now rethink my approach. – 9a3eedi Feb 18 at 10:12
95

I think you are approaching this problem from the wrong angle.

Better let the generator place a clear and visible comment at the beginning of the C header file like

// This file is autogenerated, don't change it manually,
// any manual changes will get lost after next regeneration.

then make generating the C file from the CSV file part of the build process (which you describe in the make file).

If someone ignores the comment at the beginning - bad luck, they will surely not do this a second time after loosing a few hours work.

Some additional recommendations from the commenters below (thanks to all contributers):

  • add a note to the generated comment which tool generated this file from which source

  • make the generated file read-only (and make sure the team does not use an IDE which ignores the read-only flag)

In case the header file contains parts which have to be maintained manually from time to time, then move them to a second file which is included from the generated one, so you have a clear separation between files which are autogenerated and files which are manually edited. By this separation there should be no reason to apply "manual changes to this header", if this is an "early phase of development" or not.

  • 11
    Very good answer. In addition to this, it'd probably also be helpful to put autogenerated things into a special "generated" or something like that directory, just to make it doubly obvious to people they're not supposed to mess with it. – Cubic Feb 16 at 18:44
  • 1
    @Cubic or make them read-only right after generating – Hagen von Eitzen Feb 17 at 1:43
  • 34
    I'd add This file was autogenerated by <tool> from <input.csv>. to the comment, so it's transparently clear how it can be generated. (If possible, I'd even add the command-line used to generate it, if it's simple enough, so it's completely obvious how to regenerate it). – nneonneo Feb 17 at 6:10
  • @nneonneo: Surely the tool knows its own command line which was used to generate that particular output? Even if complex, it can still add the command line to the output. – MSalters Feb 17 at 9:03
  • @MSalters So it's completely obvious how to regenerate [the .h file, not the command-line line command]. – wizzwizz4 Feb 17 at 13:50
41

Don't commit the generated C header file at all. In fact, delete the current file (thanks @user1936), change the script to call the header file .g.h (thanks @davidbak), and add it to .gitignore, so it doesn't get committed accidentally (thanks @cmaster).

Instead, commit the csv and python script, and add some custom step to generate the C header file at compile-time. Details on how to do that depend on your specific toolchain - whether your use make / cmake / etc.

Be sure not to run the script if not needed, otherwise you'll break incremental build and everything will be re-built every time. This is usually expressed as a dependency in make etc.

Do add comments in the generated file about how and from which source files it was generated, as suggested in @docbrown's answer and the comments to it. That will make tracing down issues easier, especially for people who'd change the file and get it immediately regenerated over.

  • 2
    This is very important. I would go even one step further and recommend adding the .h file to your .gitignore file, and commit that. That way, git will actively stop people from committing the wrong file. – cmaster - reinstate monica Feb 17 at 10:01
  • 3
    The one and only solution. If it is a generated file, generate it on each build and never check it in. – Manziel Feb 17 at 10:36
  • @Manziel: Generating a .h on each build can potentially be very expensive. On each clean build, yes. But for an incremental build? Instead, this answer says exact the opposite. Use your build tooling to determine if the .h is out of date (older), and only generate it when it's out of date. I've even gone to the trouble of checking if there's a relevant change to the .h. – MSalters Feb 17 at 16:53
  • That is what I meant to say. We have a number of JNI headers that are generated automatically if necessary. The important part is that each build on your build and test server is generating the files. If someone messes it up, tests will fail and you will notice immediately. Adding a header that says "do not mess around" does not hurt but relying on convention as the other answer suggests is a sure way into huge trouble once you regenerate the files – Manziel Feb 17 at 17:29
  • However, please do include your auto generated headers in source code releases (particularly if you’re building OSS), if they’re platform independent. It’s a huge pain in the neck to run random generator scripts in addition to a standard Make process, and it introduces an unnecessary build dependency. (For example, the vast majority of configure-based build systems use autoconf, so the configure script isn’t checked in, but it’s always included in source packages. Ditto for auto generated source and header files.) – nneonneo Feb 17 at 19:40
17

First a disclaimer: I don't think this is a good idea.

But here is one way to do it anyway:

void check_file_time() {
    if (strcmp(__TIMESTAMP__, "Sun Feb 16 19:38:35 2020") != 0)
    {
        asm("do_not_modify_this_file\n");
    }
}

This relies on a few GCC-specific tricks:

  • Non-standard preprocessor macro __TIMESTAMP__ expands to the modification time of the file.
  • GCC knows enough about strcmp() to optimize it away at compile time, even at -O0.
  • GCC allows inline assembler, but skips generating the invalid instruction if it sees it couldn't be reached.

Example of the error message produced:

$ touch -d 'Sun Feb 16 19:38:35 2020' test.c
$ gcc -Wall test.c
$ touch test.c  # Uh-oh, someone modified it!
$ gcc -Wall test.c
test.c: Assembler messages:
test.c:16: Error: no such instruction: `do_not_modify_this_file'

And if you only want a warning instead of error, you can use this variant:

void check_file_time() {
        int *do_not_modify_this_file;
        if (strcmp(__TIMESTAMP__, GENERATED_TIMESTAMP) != 0)
        {
                *do_not_modify_this_file = 0;
        }
}

test.c:17:28: warning: ‘do_not_modify_this_file’ is used uninitialized in this function
  • 4
    Version control like git does not care about timestamps, so this code will break if using that. – Thorbjørn Ravn Andersen Feb 17 at 16:21
  • @ThorbjørnRavnAndersen Hmm, true! I actually thought git would keep timestamps, as it keeps other file modes, but I guess it makes sense it won't. – jpa Feb 17 at 19:02
  • I was hoping it would cause a warning, not an error. A little reminder to the programmer you can say. – 9a3eedi Feb 18 at 10:07
  • Excellent idea in principle. Just instead timestamp, you could use MD5 hash for the file instead, grepping the line containing it away by incorporating a magic marker string, when calculating the hash. You could also put the MD5 in external file, and break the compilation from build script when there is mismatch. – FooF Feb 18 at 14:24
6

I've worked in a bunch of codebases with large quantities of auto-generated code. The amount of issues raised by folks modifying these files has been quite low, and usually quick to spot and quick to solve.

You've not given enough details about your build setup and what tools you're using. gcc might not have the tools for the job, or it might as @jpa mentioned, but you'd be tying yourself to it. But this seems like more of a build concern than simply a compilation concern - have a step in your Makefiles, gradles, or whatever where you check that the files aren't modified since they were generated.

However, I feel like this is more of an architectural/design concern than anything else. Here's what I suggest you do:

  • Add the disclaimer at the top of the generated files that people should not touch them.
  • Perhaps name the files foo.gen.h so it's clear they're not regular source code.
  • Place these files in /gen directory, next to your /src one in the repository (or equivalent). Anyway, have a clear separation between regular code and generated code folks aren't supposed to touch.
  • Store the csv files in the repository. Have the csv -> h generation as part of the build steps. If folks modify the .h files, they'll be regenerated anyway as part of the build.
  • "If folks modify the .h files, they'll be regenerated anyway as part of the build" - not necessarily. If the Makefile is correct, only a change of the .csv file will trigger a rebuild. If the .h file is modified instead, make will see that the .h file is younger than the .csv file, and will conclude that it's up to date. Only if your build system fails to check modification times will you get the file to be regenerated on each build, and in that case you are loosing a lot of performance due to fully rebuilding your app every time. – cmaster - reinstate monica Feb 17 at 9:58
  • 1
    @user253751 Please reread my citation from the answer. It's meaning is quite clear. Apart from that, build systems regenerate any files that are older than any of the files they depend on. If the .h file is generated from .csv, then the .h file depends on the .csv file, and will be regenerated if, and only if the .csv file is younger than the .h file. This is not the case if a developer manually modifies the .h file. – cmaster - reinstate monica Feb 17 at 12:19
  • Great suggestions – 9a3eedi Feb 18 at 10:11
1

Either the file is editable or it is generated. If you really want it to be both, you're going to have a bad time no matter what method you choose.

Preventing the edition of generated files is straightforward: re-generate the file as a compilation step. Make, CMake and so on are the tools of choice for this. The file of course needs to be excluded from versioning (but obviously its sources should be versioned).

Now there has to be a reason that people working on your project are choosing to edit the generated header instead of the CSV sources. Figure out why, and fix that. Maybe it's just about the convenience of not having to re-run the header generation script in which case running it as part of the build process is a good solution.

0

Use GNU diffutils or git diff (with GIT...). Use also some good build automation tool (like ninja or at least make), perhaps with ccache. You might have Makefile rules using cmp(1). Run also make -p to understand builtin GNU make rules.

You might consider build tools using contents, not modification times, for driving compilation commands. Look into scons or omake. You may want to look into gcc options like -M, use ccache and/or precompiled headers.

In RefPerSys we tried omake, but later switched to GNU make + ccache, and indeed we are generating header files. We gave up omake because nobody had time and will to study its documentation, and because omake is poorly packaged in recent Linux distributions.

0

Do you use continuous integration? If not, why not?

As an alternative, read the S.O question running a bash script from a make file.

The point is that the header file should be generated ("Just In Time") by the build process (whether Jenkins, etc or make), thus overwriting any manual changes.


[Update] if it's a one man project, as you say in a comment, then how about a naming convention on files, just add CONST somewhere in the file name?

  • 1
    I don't use continuous integration because it's currently a team of one (myself) and I simply don't have the time. Also it's embedded software which makes CI tricky. But the intention is there. – 9a3eedi Feb 18 at 10:00
  • All of my private one-man projects use CI (Jenkins ;-), but you can probably just use your makefile. Btw, if it's a one man project, then having your Python generate a very large "you have been warned !!" comment ought to be enough to save you from yourself :-) On even a naming convention on files, just add CONST somewhere in the file name? – Mawg says reinstate Monica Feb 18 at 10:03
  • 1
    I haven't had experience with jenkins and I'm expecting a steep curve till i get everything working. It's something I'm planning to learn eventually. The expectation is that more people will join the project at a later stage and perhaps that's when I'll add it – 9a3eedi Feb 18 at 10:10
  • If you are lucky, then one of them will know Jenkins :-) Actually, Jenkins is trivial; think of it as a cron job which invokes one or more scripts, with optional plugins to show pretty pictures (e.g a graph of number of warnings, with one item for each build). You can use it as a chance to run your Python scripts, do Static Code Analysis (Linting), etc It's all GUI driven & simple to set up --> – Mawg says reinstate Monica Feb 18 at 10:20
  • --> Arguably over kill for a single person, but I set mine to start when I commit to SVN (you do have version control, don't you?), Lint, Build, then run unit tests & alert me (by email) if anything fails. – Mawg says reinstate Monica Feb 18 at 10:20

Not the answer you're looking for?Browse other questions tagged or ask your own question.