0

Let me explain what I mean by locality and repetition. The current crop of configuration management tools decouple the configuration from everything else even when it is somewhat detrimental to do so. One case where it is detrimental is in application configuration. I believe that application configuration should live in the same repository as the application itself instead of some other repository or a chef cookbook or a puppet module. Better yet the actual chef cookbook or puppet module should be alongside the application and the build system can figure out how to package the whole thing so that the deployment mechanism can be as simple as possible.

One problem with this set up is that it can lead to a lot of repetition because several packages are likely to use similar or even exactly the same chef cookbooks and puppet modules but instead of being in some central location these cookbooks and modules are now duplicated in each application repository. The instinctive thing to do in this case is to factor out the commonality and put it in its own repository but I think this is incorrect because it introduces non-locality. Now to understand how the application works across its entire lifecycle you have to chase down dependencies in some other place and sometimes it is not clear at all where this other place is because of the separation between operations and product development.

So what is the correct trade-off here? I think non-locality makes things more magical than they need to be and enforcing locality leads to unnecessary repetition.

|improve this question
  • You should check out Docker. – Lawrence Aiello Feb 15 '15 at 12:47
  • 1
    You could perhaps do both. Keep all the shared scripts in a sub-repo (ie git submodule), and then bundle the actual toplevel deployment script alongside the application. – carpii Feb 15 '15 at 14:32
  • @LawrenceAiello How does Docker help? If you mean to put all the dependencies in the container then it is still non-local data that doesn't live alongside the application code that determines what happens to the code at runtime. Using docker just shifts the problem somewhere else and doesn't address the underlying problem of localizing all dependencies. – davidk01 Feb 16 '15 at 0:15
  • You say that factoring out commonality is non-locality, but you're really just talking a matter of degrees. You have no problem using Chef, or your language for that matter. Likewise, I imagine you use libraries in your software, rather than writing everything from scratch. In like manner, cookbook dependencies are just libraries used by your application cookbook. If you store your application cookbook alongside your application, and then include other library cookbooks via the metadata.rb file, you are following the same pattern as using a Gemfile, or including external libs in your app. – Tejay Cardon Feb 18 '15 at 17:03
  • @TejayCardon I agree it is a matter of degree but when things are "strongly coupled" meaning that the cookbook does not make sense on its own then that leads to non-locality because neither the application nor the cookbook make any sense when looked at in isolation. For library code the difference is that the library can stand on its own and there is a clear dependency from the applications side which means it can't work without the library. With chef cookbooks there is no such clear dependency since the application could work without the chef cookbook. – davidk01 Feb 18 '15 at 21:37
2

Chef solves this problem by using dependency management (Berkshelf) and by applying cookbook patterns. Using the Environment cookbook pattern, you are encouraged to store your cookbook WITH your application (and this does NOT result in duplicated cookbooks, because any library / base / dependent cookbooks will be resolved with the dependency management tool.)

When it comes to application deployment, your cookbook should be able to search your binary repository, find the install package, download it, and install it. Doing this allows you to correctly separate the builds of your application and of your configuration automation.

More importantly, here's what not to do:

  1. Build a docker image by hand and call it good "because Docker". Doing this is just as bad as having magical VM templates (and of course, nobody at your organization will understand how they were built, and that makes them difficult to update, etc). If you're going to use containerization, build the images with the automation.
  2. Store everything in git-submodules, or build the entire repository. Instead make sure you're using dependency management and do well-versioned releases of all your software, including your configuration automation.
  3. Overuse dependency management tools and break your code out into a million layers of abstraction. You really don't need a base, base, base cookbook. Just keep it simple with a few layers as necessary, refactor as you go.
  4. Pretend that your configuration management solution is an "install script". It's not, it's an all-inclusive way to describe the state of your servers.
  5. Build your cookbook with your application in the same build process. You do not want to do new builds of a large application when all you need is just to package up the cookbook. Instead, builds should be able to happen in parallel, especially if you have a large, distributed team.
|improve this answer
1

At least in the case of Puppet, you should Google the following:

  • Puppetfile (like Gemfile but for Puppet)
  • r10k
  • "control repo"
  • Hiera

The proper composition of a control repo to be consumed by r10k will solve 90% or better of the issues you've mentioned above. Your custom application can then be deployed with another component level Puppet module and be instantiated with all of the right data/config.

Other configmgmt tools have more or less similar tools and concepts but I cant pretend to be an expert on them.

As to Docker, you could make your deployment artifact an image built with Puppet rather than a package deployed by Puppet but you will eventually run into similar issues of providing correct settings for your environment and then you'll be looking at having to add something like etcd or Consul or ZooKeeper to the mix and...yes...that's also complicated.

Sorry I can't provide more detailed examples. Mobile. :)

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.