Let me explain my thoughts about architecture of the project I'm working on. The project code repository consist of:

  • Scrapy component - of course it serves to scrape data, process it and calculate relations between data. It populates MySQL database.
  • Django visualization component - it simply displays data stored in database using many filters.

Right now they are deployed as two separated docker containers which works fine. The idea of former collegues was to go further with split and split also code repositories.

I can see potential ability to create CI/CD per repository, so it will only run tests/lintes/checks and eventually will only deploy container which actually was modified. It won't run everything for other container which is ok(logical separation).

But because they are actually working on same database tables(Scrapy populates them, Django reads them) it looks like overkill for me. I would need to maintain two separate DB model specifications in sync in both repositories. Right now Scrapy uses Django ORM for interaction with DB.

What do you think? Do you think it's worth splitting code repository to two separated ones and keep in sync models in both of them? Or maybe not? Is there a way to trigger/run Gitlab CI/CD process for only affected container in single repository?

Thank you

  • I think the interesting part of this question, from an architectural point of view, is when to use multiple source code repositories. The part about how to conditionally trigger a CI/CD process is perhaps better suited for StackOverflow.
    – Rik D
    Apr 24, 2020 at 14:49
  • When you roll out a new version, do you always re-deploy both containers at the same tome or is there a chance/time window that a newer version of 1 container has to interact with an older version of the other (and your processes for database migrations take that into account)? Apr 25, 2020 at 10:45
  • @RikD Yes, you are right. More suitable(for Software Engineering stack) and important question for me is if I should split scraper from django in my case.
    – Bob
    Apr 25, 2020 at 19:00
  • @BartvanIngenSchenau Thanks for the question. Actually it depends what Im deploying. If I deploy some changes to DB model(which affects both scraper and visualization layer) then I deploy on both containers. After that I run Django migration(handy tool). But also quite often I deploy on single container. For example when I modify the way dashboard looks like which of course doesn't change anything in scraper container. I'm just wondering if splitting code just because of that is an overkill. Maybe you have some experience, advice how to tackle it?
    – Bob
    Apr 25, 2020 at 19:07

1 Answer 1


Splitting the repository in two with a duplication of the database models is just asking for problems. Over time, you will forget to make some changes in one of the repositories or make the changes in different ways. In the end, you will have two completely different models that may or may not map to the same table structure in the database.

If you do decide to split the repositories, make it at least 3 repositories with one repository for the common code (which includes at least the database models).

Splitting the repositories may make it easier to run separate CI/CD pipelines for the scraper and dashboard parts, but is also introduces a set of complications of its own.

Currently, you know that, if you have version 10 of the scraper and version 14 of the dashboard deployed, you need to update both to version 15 because of a database update. If the scraper and dashboard have separate repositories, their version numbers will also drift apart and it takes a bit more administration to document which versions of the components use the same database version and thus can work together.

In the end, you have to weigh the advantage of separate CI/CD pipelines against the disadvantage of a more complicated repository setup and the administration of which versions are compatible with each other. A large factor in that will probably be how often your database changes.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.