At work, we have an application that is run directly on dev machines, but deployed in Docker swarms (a QA swarm and production swarm). The code and CI/CD pipelines are all in GitLab CE.
It uses several private, internal NPM packages. We refer to these packages in the package.json
like so:
"@project/utils": "[email protected]:project/utils.git#branch"
This gives us a few problems:
- When developing locally on a feature branch, you have to manually edit the
package.json
but not check it in, and append the feature branch to the dependency URL. - Since it's a git URL, Yarn/NPM will not check and get the latest code (even if the
package.json
version is incremented). You have to delete the dependency from thenode_modules
and delete (or edit) theyarn.lock
/package-lock.json
file (though I believeyarn upgrade
does now upgrade git packages). We do have scripts to do this but, ew. - We have scripts in our CI/CD pipelines that have to "fix" the branches in the package.json for internal dependencies. This increases build time considerably, and makes it quite fragile. Basically:
yarn install --frozen-lockfile
./fixInternalPrivatePackages.sh package.json $BRANCH #where $branch can be #qa, feature-blah, etc.
yarn install #to now get any changed dependencies, sometimes modifies other unrelated deps in the lockfile
yarn run build
And then it's pushed to a Docker registry.
With this setup the lockfile sometimes gets modified, causing builds to fail. It's also a pain to manage on dev machines. Devs sometimes check-in their package.json
with the wrong #branch
, and this causes other people's builds to fail.
Possible solutions:
- Automate tagging internal dependencies, then adjust the
package.json
with these tags (could really just be the commit hash). - Private package repository. This fixes the problem with getting the latest version if the latest is always pushed. I believe this should fix the
fixInternalPrivatePackages
hacky script situation as well. - Git/NPM pre/post hooks/scripts to fix a project's own
package.json
(remove all branches, etc.)
But all these solutions just seem more like band-aids on a fundamental problem. Are we doing this the right way? Are there any example projects with internal dependencies like this that I could get some inspiration from?
I feel like one fundamental issue is that the internal dependencies need their own independent QA->Prod cycle, so that they are already locked to a specific version before the rest of the app is moved along from QA->prod.
Another problem is that while yarn and npm now have lockfiles (we don't talk about the time before lockfiles), our workflow breaks this functionality when it has to modify the package.json
to switch branches, and reinstall dependencies without the --frozen-lockfile
switch. We want reproducible builds, and I can't guarantee that we have them.
It takes a long time to get a developer up to speed on our project because of all these configuration management issues. The actual code itself is much simpler than the code that requires DevOps knowledge to get it running.