How to apply DRY to files shared by repositories?

Question

I've got a few files which are used in several of my repos:

• functions.sh, shell library to for example print a colored warning/error message or the documentation of a script file.
• Makefile; a standardized one which installs the file $(CURDIR)/[dirname].sh to $(PREFIX)/[dirname] and references a test script.
• LICENSE.
• tools.mk; Makefile commands to for example print all the variable definitions in the parent Makefile.

These are more or less stable, and some are used in probably over a dozen repos. I've been thinking how to keep this DRY, but none of the options so far seem satisfactory:

1. Keep doing it like now, creating a copy for each new repo. This keeps the code together with all its dependencies (avoiding bugs when the general solution is not general enough), but changes which are applicable in multiple places have to be applied in each separately.
2. Add an executable to each of the repositories to download the files needed. This means that developers and end users will have to run an extra command to get all the relevant files, and it breaks the possibility for developers to modify and commit/push the included files.
3. Use git submodule or equivalent. This at least keeps the repositories connected, but in the Git case it seems like it's restricted to "a dedicated subdirectory", so no top-level files, and no mixing with parent repository files which belong in the same directory. This could be circumvented with symlinks, but that's an ugly workaround for the obvious ideal situation.

The ideal solution should:

1. Communicate with the correct repository when doing an operation on a file.
2. Allow includes in the same directories as the parent directory.
3. A single, simple command should be enough to update the entire repository and all includes, no matter how many or how deeply nested.
4. Allow includes in the top-level directory.
5. Not incur significant developer or user constraints (must be online while installing) or extra work (running a "pre-install" command separately from the "install" command).
6. Allow "cherry-picking" of files to include. Many projects might need a different Makefile, for example, and including one which is not used is just ugly (and would get uglier as more files are added).

Is this sort of thing possible with current software?

Answer 1

In SVN the concept is called "vendor branches", ie you think of your shared files in much the same way as you think of a 3rd party library that you're using. You make a 'read-only' copy of the external files to your repo and use them. but never change them - that only occurs when you re-import the vendor files.

If you do make changes to a shared file in a project, you can (by branching it inside your repo). When you get a new import, you need to merge the changes again.

Answer 2

If files themselves put-together are independent (rather self dependent) - it does make sense to have them in separate repo and treat it as a dependency project for a give repo. This will minimize the copying files.

However, from the example you gave - these are things like Makefile, license etc. are not full fledged module by themselves. If you keep them separate - this will be random amalgamation and wont help as much as the pain it will put you in.

Also, given the same examples, i also don't see that to be a great resource saving exercise either. So overall, i don't quite see much value in this exercises.

I am not saying anything against DRY as a principle; but DRY or NOT DRY - will it have any significance here?

Answer 3

One option is to have a separate repository for the shard files and then all repositories that use those files either fork from or merge in the shared file repository.

Here's an example with mercuiral. Say I've got a repo shared with the shared files and a repo project which has a project that uses the shared files. To get the shared files into the project repo I run the following commands from the project repo:

hg pull -f /path/to/shared/repo
hg merge
hg ci -m "merged shared repo"

This pulls and merges everything from the shared repo into the project repo. I can also make any changes to the files merged from the shared repo (say a specific makefile rule) and those changes will remain local to the project repo.

Now say I've updated the shared repo with some new features or bug fix and want to include that in my project repo. Then from the project repo I simply run

hg pull /merge/to/shared/repo
hg merge
hg ci -m "merged updated shared files"

And I have the updated versions. No -f flag was needed on the pull this time since the repos are now related.

For git the commands are almost identical. To merge the shared repo with the project repo I simply run

git fetch /path/to/shared
git merge FETCH_HEAD

git also supports remote repos making it so I don't have to enter /path/to/shared whenever you want to merge. I can add shared as a remote repo with

git remote add shared /path/to/shared

Then whenever I want to merge with it I enter

git fetch shared
git merge shared/master

Where master is the name of the branch in the shared repo.

While this isn't a single command, I don't think two commands is so bad (the commit form mercural doesn't count :P). Also it allows you to easily "cherry-pick" files since the files merged in from shared are like any other tracked files in the project repository. The downside is it doesn't support requirement 1. If I change the shared repo's files in the project repo, these changes will not be reflected in the shared repo. Instead I have to change them in the shared repo and merge those changes in.