Proper development workflow for fullstack app with multiple developers?

Question

Say that I am developing a web application that has the following structure:

• An SPA web frontend (angular in my case)
• Postgresql database with:
  • A bunch of initial data in CSV's and JSON's that need to be loaded.
  • A bunch of cron-job scripts which periodically fetch data from external sources and feeds it into the database.
• A thin 'middleware' web server which provides a GraphQL API to the database (as well as auth).

Currently, I develop this app with all three parts running separately... I run the frontend using Angular's dev server, I run the middleware as a standalone process (with nodemon), and I have a development database against which I run scripts manually.

This works "ok" as long as I am working solo, but it is quickly become unmanageable as I try to bring on other frontend devs (I will still be the only middleware/database guy). For example, I have put instances of the middleware and database on a development server that's accessible to all. However, this makes it hard for me to make changes (the data model and API are changing rapidly) because it may break whatever the others are working on. I think I need to be able to version the api and database, but I'm not exactly sure how to do that. Also, there may be problems with version skew between the components.

One thing I've thought about is putting the middleware and database (with all data preloaded) into a docker container and having the other frontend devs run it with docker compose or something. However, I'm not sure how well that would work on windows. I don't have the expertise to run something more complicated like kubernetes at the moment. Also, in the long term, I'm not sure I want to deploy in containers because putting a database in docker does not seem 'right' to me for some reason (maybe I'm biased).

Any advice on the correct development workflow and/or project structure and/or products and services that might help?

Answer 1

You definitely want to figure out way to make it easy to run the whole solution locally on development machine.

You mention "I'm not sure how well that would work on windows". Maybe you want to limit all your developers to only use Linux? That would simplify things greatly. Then, you can have bunch of bash scripts that setup everything locally for development.

You mention Docker. Docker on Windows can easily run Linux images. So it might simplify deployment and running of pieces of your solution locally.

I recommend to run an experiment: Completely clear your development environment. Clean OS installation. Check out code from source control. After that, how long does it take to setup your development environment and how much of that is manual work? In ideal situation, you should run single batch script and it should setup your whole environment. It will setup all the services, build the source code, run automated tests, maybe even sets up IDEs with reasonable team-wide defaults.

Answer 2

We have been using docker for development and testing and it works great. The docker-compose files are stored in git as are all of the set up/configuration scripts so everyone is working with the same setup. Maven has plugins to run containers as part of automation integration tests which makes life a lot easier.

NOTE: Be sure to spec explicit versions of the docker images you are using; don't just use the "latest" tag.

You do not need to think about kubernetes during development.

Whether you deploy production using containers or not does not affect day-to-day development: your software doesn't care if the other end of a network connection is running in a lightweight container, full-on VM or bare-metal machine.

For what it's worth, I think you're biased against containers for some reason; we've run databases in containers with and without kubernetes with no problems. The data has to be stored in a persistent volume, but that's easily configured.

Answer 3

Say there is a branch named develop which developers integrate with frequently. I usually find it valuable to have a CD process deploy develop builds to a shared machine that's accessibly by all. That CD process needs to be protected by a quality gate, for example we usually don't allow merges into develop if unit tests fail and integration test will run after develop has been deployed.

Depending on your development velocity and frequency of integrations, develop should remain stable enough for other feature development to occur against that. By that I mean front end can test a feature branch against the state of back-end that's made it into develop, and back-end can test a feature branch against a state of front-end that's in develop.

For breaking changes, feature flags are a useful capability to have. This way one team can work on a breaking change and turn that on based on their needs while everyone else remains unaffected.

For APIs, API fakes can be really helpful. API fakes are dumb servers that spew out pre-defined content with no business logic. I think API fakes are frequently used to test Angular code with no back-end dependency.

Answer 4

For team workflow you typically want to have a good git strategy for the team to ensure time spent committing and maintaining the software is mostly spent on productive activities like the velocity of feature changes and not spent on merge conflicts and rebasing.

I.e. to have a good team workflow you require the best git workflow for the team. To choose the right git workflow you need to consider how you will deploy the application/s.

You can chose 1 git workflow and force everyone to use it on every repository (violating your main goal of having a good team development workflow)

Or you can do a git workflow for each repo based on how that application is deployed, i.e. identify the release and deployment strategy first to then decide on the git workflow appropriate.

You've identified 3 high level projects, but are these only 3 repositories? Maybe yes but modern teams follow a microservices architecture meaning these 3 projects will require 10s or even 100s of small repos just for these 3 projects.

i.e. There is specific release strategy called semver (semantic versioning) which requires you to progress code changes on at least 3 current milestones simultaneously;

1. major
2. minor
3. patch

In practice that means the git workflow cannot be pure 'mainline' or the trunk-based git strategy (that never really exists anyway, they're usually mainline and just called trunk from a lack of understanding). Because if you release major version 1.0.0 on the 'main' branch and need to patch 0.9.1 to become 0.9.2 then a rebase of 'main' branch to fix the point in the history where 0.9.1 was will actually effect the current 1.0.0 (which is not going to work) so mainline cannot work with SemVer and typically you will want to use SemVer for anything you publish to a package repository OR the repo is used as a dependency (internally or externally) by other repositories (ergo microservices) then 'mainline' and 'trunk' are not an option becuase they are not git compatible with the semver release strategy.

Another release strategy (or rather a deployment strategy) may be for the 'middleware' you described as GraphQL API. These typically need to be deployed in self-contained stages representing a stable API that will be replaced by the latest stable API; i.e. popular deployment as;

• Canary
• Blue green
• feature flags

There are many more, but these are 3 I see most.

For these to work you will need the git repo support deployment 'release' branches. Long lived branches that are the equivalent of the 'main' branch.

So basically work backwards and decide on the deployment strategy needed for each repo first, use that to decide a git strategy per repo, then get developer workflows setup to support these and they'll be very productive developers who spend far less time rebasing and dealing with merge conflicts.

A rule of thumb is if you spend more than a few seconds on a merge conflict in a day, or do more than 1 rebase per year, your doing git wrong. I work across 10s organisations and contribute to 20+ repos per month and the healthiest of these know that a rebase or a hard merge conflict to resolve is an unhealthy team and a sign of poor git strategy for their deployment and actually change the git strategy to be better suited to how they release code.