How to version generated code without checking them into VC?

Question

I have written a code generator for my company, that basically has the following workflow:

                 entity xml definition  
                         |
                         v
   gets validated and put into java runtime classes by jaxb
                         |
                         v
runtime classes are written to .java files with stringtemplate

Now it is possible to start a server and use those files. So far, so good.

Before we can install the server for our customer, it needs to be validated for GAMP 5, which is a lot of work. Every time there is a change to the system, we have to test EVERYTHING from scratch, if we cannot make sure, that it has not changed.

Therefore, I need to have a concept on how to version the generated code.

The problem (1) is, that the generated code is neither commited to git, nor to SVN; only the xml definitions are put into SVN.

Another thing (2) is, that the once the gradle task to generate the code, which is necessary as soon as something changed to the xml files, (currently) re-generates ALL .java files, even if there was no change to the xml file of the respective entity.

The third (3) thing to have in mind is, an entity can be extended by another xml file. the workflow for this would look like

                 entity xml definition  extension xml definition
                         |                         |
                         v                         v
   gets validated and put into java runtime classes by jaxb
                         \                        /
                          \                      /
                     Merging the runtime classes together
                                     |
                                     v
      runtime classes are written to .java files with stringtemplate

An Idea

was to generate the .java file, and before overwriting old files, read them, and only overwrite them, if they differ. The output message would then say, which parts of the software needs to be re-validated

Another approach would include generating hashcode for all the files and see if those differ from previous hashes.

What would be the best way to do so?

Answer 1

It is generally considered a bad idea to version-control generated files. The problem is that you most likely already have the generating files stored in the version control system. Then you have essentially the same information twice. Also, somebody can accidentally forget to update the generated files after updating the generating files.

Have you considered an intelligent build system that automatically generates the generated files? If your tools always modify the timestamp of all generated files despite the fact that they may not be changed, the solution is to compare the contents of the new files with the contents of the old files, and updating them if they differ. Don't just use hash code, it's way too risky. Hash codes are meant for hash tables and there will be collisions in this case. Instead, use a secure hash such as SHA256 (SHA1 is already broken, and MD5 has been broken for a long time). Or you can do full file comparison instead of secure hash.

Or, you can update your tools instead of doing the hacks to prevent modification in your build system.

Anyway, since you're using Java, the compilation will be fast even if everything needs to be recompiled. My current project uses C and full compilation takes minutes (a ten-liner C file containing few #includes can end up being parsed as a ten-thousand-liner C file...).

Answer 2

As I commented, I would suggest to version control the generated files.

You'll put them in a (version-controlled) subdirectory, perhaps generated/

You'll add files in generated/ only after checking that the generated file content actually have changed (its time stamp is not enough).

It is a common thing for many bootstrapped compilers (which version control some "compiled" form, e.g. to be able to compile themselves), including Ocaml in boot/ocamlc, my GCC MELT (which version controls generated C++ code in generated/), Bigloo, Chicken Scheme

^{(an interesting exception is Rust: it does not distribute the executable Rust compiler, but the installation script is fetching -using wget or curl command- that executable from a canonical remote URL)}

You do need to write specific scripts to install the generated files in the version controlled subdirectory.

You may want to use hooks of your VC system, e.g. git hooks if using git (or perhaps svn hooks for svn, but I leave you to check that; I never used them)

You probably should take care of when you actually put new versions of your generated files in your generated/ subdirectory. Be careful when doing this (you probably don't want to do that too often, to avoid useless commits; but you surely want to run a lot of tests before committing that).

Answer 3

We came up with a totally other opportunity:

Instead of versioning the generated code, we will version our whole buildsystem and the xml files.

This way, we can definititely say, with what version of the generator the code was generated, and what version of xml files was used.

Additionally, we dont need to store all the generated code in version control systems.