1

At our job we are having a heated discussion with the DevOps team on how to architect and modularize our microservices. We come from a 2-coupled-monoliths approach, so we are trying to avoid common pitfalls.

Our CI/CD is done with GitHub Actions, and we use Kubernets to deploy our applications.

We started a microservice a few weeks ago, and my development team basically had:

  • An API
  • A Relational DB
  • Some background jobs for recurrent operations or to handle some heavy load
  • A Redis DB to store the background jobs

The DevOps team refuses to support these kind of applications, claiming that:

  1. The migrations of the database should be in a separate repository, because this helps to separate concerns.
  2. Also, having the migrations on a separate repository helps them to deploy without downtime.
  3. And last but not least, that our approach is a bad practice from "monolithic frameworks"
  4. Database changes can be deployed independently from app code by reducing coupling

Our problem as a dev team, is that when you use an ORM the application is tightly coupled to the database, so db migrations and app code arent really independent and decoupled at all. Wheter the migration is close to the application code or separated from it, the developer must always think on how not to break the application when running these migrations. Plus, we are a small dev team (3 now, probably 6 by end of year) so the same person that is touching the app code will have to write migrations, its not that you have a DBA to guard for these kind of problems.

This is one discussion. The other one is, again, that DevOps says that background jobs should be independent and have their own data models. Again, I disagree too much with this point of view, because it just brings up potential duplication of the code just to satisfy someone else.

My concern, as a developer lead, is that we are moving towards a distributed monolith.

Am I just plain wrong? I googled, read blogs, even read a bit about k8s and my conclusion is that the DevOps team is just refusing to cooperate with us, because AFAIK there can be done zero downtime deployments with k8s if the migrations are in the app repository or not.

Improve this question
5
  • 1
    Sounds like your DevOps team is just the "Ops" team :) If you have a separate, dedicated DevOps team that you have to engage into a power struggle with, that's in no way DevOps, despite the name. Not a super helpful comment, but puts thing into perspective. I guess you either work out your differences with the Ops team, or you present your case and convince the decision makers to let you take the ownership of the Ops part of your application for yourselves (e.g. maybe explain how going down the other path will hurt their business). Either way, there's going to be negotiation involved.
    – Filip Milovanović
    Commented Mar 27, 2022 at 4:52
  • I think another issue may be having a deployment process with a dependency on the Git repository structure. I'd typically expect separate pipelines for CI and Deployment, with CI merely publishing one or more built, packaged, versioned artefacts, and a separate, independent deployment pipeline responsible for deploying whatever artefacts are necessary into an environment. If your deployment process only uses built, published artefacts rather than git repositories then the deployment pipeline should be agnostic to the source project structure in git.
    – Ben Cottrell
    Commented Apr 25, 2022 at 20:19
  • I think you nailed it in this part. Our deployment pipeline is tightly coupled to each repository using Github Actions. I am not exactly sure the motivation of the devops team to do that rather than abstracted solution from the repositories. Maybe it was just "being used to it" or not wanting to move away from it to justify the "bad move". Either way, we exposed our arguments and realized that there was no clear benefit at least in the very short term given the solutions we are building.
    – Matias
    Commented Apr 26, 2022 at 21:37
  • Does your DevOps team support other applications? Are they just asking you to do the same thing as everyone else? Or is this new to all of you?
    – John Wu
    Commented Apr 28, 2022 at 8:23
  • Hey I've been facing the same problem in my company lately and the direction I'm investigating towards is Views. Instead of talking with the actual DB which can be a single common instance, each microservice interacts with Views that reflect your ORM models and sync themselves with the underneath data for the actual DB. This way you can use ORM to couple with a View, but Views are decoupled from the underlying DB.
    – Carlo Moretti
    Commented Aug 25, 2022 at 14:50

1 Answer 1

Reset to default
1

Who’s the architect?

It isn’t DevOps job to design the solution. The architect must design something that will get the support it needs. Be grateful the DevOps team is being clear about what they will support. Be equally clear to your boss about what this refusal is going to cost in development.

It may be that the DevOps team is just pushing the work around to make their job easier. But it could be they have a point and you have some new techniques to learn. Stay open to that possibility.

Improve this answer
2
  • The architect is on my side but he says we must come up to a common field rather than disagreeing for the sake of the team. I am trying to keep an open mind but I dont get how this works with ORMs where the migration system is baked in or generates automatic migrations. This kind of pushes the dev team to have a standalone library just where the ORM entities live + the migration scripts, and that being added from multiple projects. For me this is just more overhead for a really small team, and if it grows now you depend on proper communication so nothing is fucked up during release management.
    – Matias
    Commented Mar 27, 2022 at 0:13
  • All I can see clearly is that there is more going on here than you are explaining. There is no good canned answer. But so long as you can’t see how your team can do this work, with these restrictions, in this amount of time, don’t be quiet about it. Be willing to learn. But don’t give in to a death march.
    – candied_orange
    Commented Mar 27, 2022 at 3:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.