We ran into an unfortunate situation at work recently and I've been wondering what we can do to avoid similar problems in the future.
We make embedded systems. The FPGA code is in one SVN repository while the firmware & software code is in a different repository. Building the firmware requires the built FPGA images, so these binary files are also stored in the firmware/software repo. The FPGA build can take 2-3 hours and may randomly fail. To my understanding, these late failures occur because of the complex timing issues in the FPGA logic. Developer errors such as bad syntax will cause the build to fail within minutes.
Because the builds take so long, the FPGA developers generally make their changes, build, give the new image to the firmware developers, and then commit to SVN. A few weeks ago, this caused a lot of issues as we were preparing for a release. One FPGA developer made a small change and started a build. Another FPGA developer made a commit (after he had been told that his changes did not need to be included in this release and that he should wait to commit). After this commit, the initial build failed. The first FPGA developer then had to revert the other's changes, commit his changes, and spend a significant amount of time inspecting the code in SVN to make sure it reflected the code used to build the image required for the firmware build before we tagged for the release.
If we were using git or another DVCS this wouldn't be a problem, since each FPGA developer could commit locally before building and push to a server after a successful build. Switching our version control is currently not an option. We're in the process of throwing more hardware at the problem of long FPGA builds and we're investigating whether the build time can be improved by changing settings in the build tools.
Is there anything else we can do to improve our processes so we don't repeat this situation or discover other problems with versioning and conflicts?