4

Imagine you have the following scenario. Your organization deploys some app core to its mission. For example if you're Stack Exchange, this might be the customer facing web server. Let's also say you subscribe to something like the 12-Factor App philosophy. Each app mantains its own SCM repo, which acts as the canonical source of truth. You might also have org-wide libraries which are consumed by zero, one or multiple apps, and maintain their own SCM repos. Finally the app itself may be deployed over many different sites, each with potentially different configurations or underlying infrastructure.

Okay, now the wrinkle comes with how you handle tooling that's designed to monitor, analyze or detect anomalies with the actual app production runs. The purpose might be for sysadmin reasons (checking for critical error messages in app logs), data science (post-hoc analysis of user behavior), performance profiling, or some other reason.

What are best practices for organizing this tooling in your organization's multi-repository structure.The key question is do you put this tooling with the app's main repo? (E.g. add a new top-level directory called tools/ to the repo frontend-web-server). Do you separate all the tooling for a given app into a single separate repo? (E.g. new repo called frontend-web-server-tools) Or create a separate repo for each specific tool? (E.g. frontend-web-server-user-analysis, frontend-web-server-runtime-monitor, frontend-web-server-profiler)

I see some strong arguments on all sides:

  • The tooling may have fairly strong dependencies on the underlying app code itself. E.g. if we change the app's log format, then anything consuming those logs probably also has to be change. Keeping it a single repo reduces the chance of breaking the tooling on codebase changes.
  • The tooling itself probably conforms to much looser software engineering standards, and a lot of tools may created on a one-off or ad-hoc basis then thrown away. We may not want to pollute the main repo with this.
  • Some of the tooling may span multiple apps. For example for some data science analytics script we may want to join activity in the front-end app with data from a backing microservice.
  • The tooling itself is likely to benefit from a shared codebase. If we have a tool to do X and a tool to Y, they may both need some logic to do Z. Obviously DRY, so we often we probably want similar tools to at least be in the same repo with each other.
  • The tooling may also consume code directly from the app's codebase. Maybe we have a tool that replays activity for some test environment. That might be a lot easier to do by relying on a single module within the app to avoid DRY. Compiling the tool along with the app's codebase is a lot easier if they share a repo. It seems impractical to fork an entirely separate library repo just for one repo.
  • Some of the tooling might be specific to the deploy. You wouldn't want to pollute the app codebase with deploy details.

Maybe a single approach doesn't capture all the idiosyncrasies of different plausible scenarios. But how do large organizations typically handle situations like this? Or at least how do they approach making these decisions?

1

Your whole question can be answered by changing the point of view and not seeing those "tools" as something separate from the main application, but as a part or feature of the application (even if they are not features for the end user, but features for the devs, admins or testers). Then the rules for managing the tools are not different from the rules for any other feature or component inside your app.

Let me demonstrate this by commenting to your points one by one:

The tooling may have fairly strong dependencies on the underlying app code itself. E.g. if we change the app's log format, then anything consuming those logs probably also has to be change. Keeping it a single repo reduces the chance of breaking the tooling on codebase changes.

Tools which have a strong dependency on the app code can typically not reused outside the context of the app, so they should be part of the app's codebase and repo. So make a clear distinction between such tools and other ones you may reuse elsewhere.

The tooling itself probably conforms to much looser software engineering standards, and a lot of tools may created on a one-off or ad-hoc basis then thrown away. We may not want to pollute the main repo with this.

I would not see it as a problem when throw-away tools get it into the main repo for a short time - if they are really throw away tools, they won't stay long there, right? However, if they turn out not to be thrown away, and need to be maintained over a longer period, making them conform to your standards makes sense. But if your software engineering standards don't fit to such such tools or "internal application features", then you might rethink your standards.

Some of the tooling may span multiple apps. For example for some data science analytics script we may want to join activity in the front-end app with data from a backing microservice.

Your link to the 12-factor-app tells it all: it recommends to factor shared code into libraries which can be included through the dependency manager. And shared code for tools is in no way different from any other reusable code.

The tooling itself is likely to benefit from a shared codebase. If we have a tool to do X and a tool to Y, they may both need some logic to do Z. Obviously DRY, so we often we probably want similar tools to at least be in the same repo with each other.

You can put generic, reusable tools into a separate repo when they cover the same thematics (just like you should not put arbitrary things into a library). But if those tools X and Y are app-specific tools, and Z is a shared tool/component, X and Y should be in the app's repo, and Z in its own.

The tooling may also consume code directly from the app's codebase

If that means the tool has to be modified specificially to the app, then it belongs to the app's codebase. If that leads to a situation where an entire reusable library needs to be forked/copied, then there is something wrong with the way the library can be configured. There should be a strong separation between the generic, reusable part (which is kept in the external repo) and the configuration part which is specificially created for the app (and this config should be in the app's repo).

Some of the tooling might be specific to the deploy. You wouldn't want to pollute the app codebase with deploy details.

Tools for deployment of a specific app belong to that app, I don't see this as a "pollution of the codebase", quite the opposite. Of course, deployment tools should have their own folders within the folder structure of the codebase.

  • Thanks! This approach seems to be sensible and well thought out. It cut through a lot of the ambiguity I had. – user79126 Mar 5 '18 at 8:28

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.