1

I have written a code generator for my company, that basically has the following workflow:

                 entity xml definition  
                         |
                         v
   gets validated and put into java runtime classes by jaxb
                         |
                         v
runtime classes are written to .java files with stringtemplate

Now it is possible to start a server and use those files. So far, so good.

Before we can install the server for our customer, it needs to be validated for GAMP 5, which is a lot of work. Every time there is a change to the system, we have to test EVERYTHING from scratch, if we cannot make sure, that it has not changed.

Therefore, I need to have a concept on how to version the generated code.

The problem (1) is, that the generated code is neither commited to git, nor to SVN; only the xml definitions are put into SVN.

Another thing (2) is, that the once the gradle task to generate the code, which is necessary as soon as something changed to the xml files, (currently) re-generates ALL .java files, even if there was no change to the xml file of the respective entity.

The third (3) thing to have in mind is, an entity can be extended by another xml file. the workflow for this would look like

                 entity xml definition  extension xml definition
                         |                         |
                         v                         v
   gets validated and put into java runtime classes by jaxb
                         \                        /
                          \                      /
                     Merging the runtime classes together
                                     |
                                     v
      runtime classes are written to .java files with stringtemplate

An Idea

was to generate the .java file, and before overwriting old files, read them, and only overwrite them, if they differ. The output message would then say, which parts of the software needs to be re-validated

Another approach would include generating hashcode for all the files and see if those differ from previous hashes.

What would be the best way to do so?

  • 1
    Well, your first idea seems fine, I did something similar in the past with generated code to avoid changing the time stamp of a file when nothing has changed. Why not give it a try and see how it works? – Doc Brown Mar 1 '17 at 7:45
  • @DocBrown Because I am in my second year junior software developer and I have never done any versioning before. I do not just want it to 'somehow work', but want it to 'work well'. Regarding design patterns, where you want to apply patterns, that are proven to work well, I was looking for a way, that is also proven to work well :) – Do Re Mar 1 '17 at 7:51
  • @DocBrown It would also be nice to find a way, where I would not have to go through all steps before deciding to throw everything away, as nothing changes – Do Re Mar 1 '17 at 7:53
  • 1
    Did you consider version controlling the generated files in some generated/ subdirectory, and you'll put files there only when their content has changed. I do that in GCC MELT – Basile Starynkevitch Mar 1 '17 at 9:19
  • 1
    (1) Can you map a generated file back to the XML file(s) that are its source? If so, you can write a makefile-like rule; I suppose Gradle supports dependencies like that. (2) Note that the generated files don't only depend on the XML files, but also on all the software used in the intermediate steps, which also can change. (3) The generated files are good to have around, but essentially their storage is a persistent cache. It could live somewhere else, where other build artifacts live. – 9000 Mar 1 '17 at 21:58
4

It is generally considered a bad idea to version-control generated files. The problem is that you most likely already have the generating files stored in the version control system. Then you have essentially the same information twice. Also, somebody can accidentally forget to update the generated files after updating the generating files.

Have you considered an intelligent build system that automatically generates the generated files? If your tools always modify the timestamp of all generated files despite the fact that they may not be changed, the solution is to compare the contents of the new files with the contents of the old files, and updating them if they differ. Don't just use hash code, it's way too risky. Hash codes are meant for hash tables and there will be collisions in this case. Instead, use a secure hash such as SHA256 (SHA1 is already broken, and MD5 has been broken for a long time). Or you can do full file comparison instead of secure hash.

Or, you can update your tools instead of doing the hacks to prevent modification in your build system.

Anyway, since you're using Java, the compilation will be fast even if everything needs to be recompiled. My current project uses C and full compilation takes minutes (a ten-liner C file containing few #includes can end up being parsed as a ten-thousand-liner C file...).

4

As I commented, I would suggest to version control the generated files.

You'll put them in a (version-controlled) subdirectory, perhaps generated/

You'll add files in generated/ only after checking that the generated file content actually have changed (its time stamp is not enough).

It is a common thing for many bootstrapped compilers (which version control some "compiled" form, e.g. to be able to compile themselves), including Ocaml in boot/ocamlc, my GCC MELT (which version controls generated C++ code in generated/), Bigloo, Chicken Scheme

(an interesting exception is Rust: it does not distribute the executable Rust compiler, but the installation script is fetching -using wget or curl command- that executable from a canonical remote URL)

You do need to write specific scripts to install the generated files in the version controlled subdirectory.

You may want to use hooks of your VC system, e.g. git hooks if using git (or perhaps svn hooks for svn, but I leave you to check that; I never used them)

You probably should take care of when you actually put new versions of your generated files in your generated/ subdirectory. Be careful when doing this (you probably don't want to do that too often, to avoid useless commits; but you surely want to run a lot of tests before committing that).

  • Do you know, if there is something similar to Git hooks for SVN`? – Do Re Mar 1 '17 at 15:30
1

We came up with a totally other opportunity:

Instead of versioning the generated code, we will version our whole buildsystem and the xml files.

This way, we can definititely say, with what version of the generator the code was generated, and what version of xml files was used.

Additionally, we dont need to store all the generated code in version control systems.

  • Isn't this what juhist answered? Shouldn't you accept their answer? – DisgruntledGoat Mar 3 '17 at 11:36
  • @DisgruntledGoat Yeah, kinda. you are right. Gave him the upvote – Do Re Mar 3 '17 at 14:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.