15

At work, we have an application that is run directly on dev machines, but deployed in Docker swarms (a QA swarm and production swarm). The code and CI/CD pipelines are all in GitLab CE.

It uses several private, internal NPM packages. We refer to these packages in the package.json like so:

"@project/utils": "[email protected]:project/utils.git#branch"

This gives us a few problems:

  1. When developing locally on a feature branch, you have to manually edit the package.json but not check it in, and append the feature branch to the dependency URL.
  2. Since it's a git URL, Yarn/NPM will not check and get the latest code (even if the package.json version is incremented). You have to delete the dependency from the node_modules and delete (or edit) the yarn.lock/package-lock.json file (though I believe yarn upgrade does now upgrade git packages). We do have scripts to do this but, ew.
  3. We have scripts in our CI/CD pipelines that have to "fix" the branches in the package.json for internal dependencies. This increases build time considerably, and makes it quite fragile. Basically:
    1. yarn install --frozen-lockfile
    2. ./fixInternalPrivatePackages.sh package.json $BRANCH #where $branch can be #qa, feature-blah, etc.
    3. yarn install #to now get any changed dependencies, sometimes modifies other unrelated deps in the lockfile
    4. yarn run build

And then it's pushed to a Docker registry.

With this setup the lockfile sometimes gets modified, causing builds to fail. It's also a pain to manage on dev machines. Devs sometimes check-in their package.json with the wrong #branch, and this causes other people's builds to fail.

Possible solutions:

  1. Automate tagging internal dependencies, then adjust the package.json with these tags (could really just be the commit hash).
  2. Private package repository. This fixes the problem with getting the latest version if the latest is always pushed. I believe this should fix the fixInternalPrivatePackages hacky script situation as well.
  3. Git/NPM pre/post hooks/scripts to fix a project's own package.json (remove all branches, etc.)

But all these solutions just seem more like band-aids on a fundamental problem. Are we doing this the right way? Are there any example projects with internal dependencies like this that I could get some inspiration from?

I feel like one fundamental issue is that the internal dependencies need their own independent QA->Prod cycle, so that they are already locked to a specific version before the rest of the app is moved along from QA->prod.

Another problem is that while yarn and npm now have lockfiles (we don't talk about the time before lockfiles), our workflow breaks this functionality when it has to modify the package.json to switch branches, and reinstall dependencies without the --frozen-lockfile switch. We want reproducible builds, and I can't guarantee that we have them.

It takes a long time to get a developer up to speed on our project because of all these configuration management issues. The actual code itself is much simpler than the code that requires DevOps knowledge to get it running.

4
  • From your problem description it seems like the app and the dependencies are always developed together, so that work on a dependency requires changes to the application build process. This suggests such dependencies should not be treated as a separate package/repository – they are part of the same codebase. But later you note that dependencies have an independent lifecycle. Could you clarify your development approach, e.g. explain why a monorepo would not be appropriate?
    – amon
    Aug 16, 2018 at 6:31
  • 1
    @amon, The project started as a monorepo (in TFS), but build times were atrocious (~30-45 mins). Though that mostly was a function of the slow TFS build server. We split it out to 2 APIs, 2 Apps (front ends), and 4 less often updated dependencies (translations, utils, configuration, common UI components). This more-or-less mirrors the Docker service setup (2 APIs, 2 Apps, +DB), but otherwise was a matter of personal preference at the time.
    – Nick
    Aug 16, 2018 at 13:04
  • @amon ...but now that you bring it up I suppose we could migrate to a monorepo. The dependencies are tightly coupled. I need to see how well NPM/yarn/our build handles local dependencies.
    – Nick
    Aug 16, 2018 at 13:07
  • Since this question was asked, Gitlab has made the built-in NPM Registry feature available in Gitlab CE, which substantially simplifies this issue: each package can have a CI job that publishes a package to the internal registry, each other job can just npm-install it without having to use a Git URL.
    – amon
    Jan 12, 2021 at 12:54

2 Answers 2

1

We used the .npmrc file to define multiple repositories with different logins. (works for yarn too)

That means we have for example two dependencies, a public hosted one and a private hosted one. Important, the private one is online hosted, just password protected, so that only authorized people can access it.

package.json

"angular/angular-core": "1.0.0"  
"@MyCoolProject/utils": "1.2.3"  

Then we added a .npmrc file to our project (here an example for an Azure DevOps repo)

@MyCoolProject:registry=https://pkgs.dev.azure.com/my-organisation/my-project/_packaging/some-azure-feed/npm/registry/
always-auth=true
; begin auth token
//pkgs.dev.azure.com/my-organisation/my-project/_packaging/some-azure-feed/npm/registry/:username=my_username
//pkgs.dev.azure.com/my-organisation/my-project/_packaging/some-azure-feed/npm/registry/:_password=a_base64_encoded_password_or_token
//pkgs.dev.azure.com/my-organisation/my-project/_packaging/some-azure-feed/npm/registry/:[email protected]
//pkgs.dev.azure.com/my-organisation/my-project/_packaging/some-azure-feed/npm/:username=my_username
//pkgs.dev.azure.com/my-organisation/my-project/_packaging/some-azure-feed/npm/:_password=a_base64_encoded_password_or_token
//pkgs.dev.azure.com/my-organisation/my-project/_packaging/some-azure-feed/npm/:[email protected]
; end auth token

That way all dependencies starting with "@MyCoolProject" are downloaded from Azure, while the rest is downloaded from the default repo.

The .npmrc file could be set per project or also per user (depends on where the file lives). So you can have user specific login credentials without having to change a file that gets checked into git.

This works for our local dev machines and also for the CI/CD pipelines.

0

Well, I stumbled on Lerna from a post from Hacker News and it appears to be just what we need. I will accept if it works out but it's going to be a while.

Splitting up large codebases into separate independently versioned packages is extremely useful for code sharing. However, making changes across many repositories is messy and difficult to track, and testing across repositories gets complicated really fast.

To solve these (and many other) problems, some projects will organize their codebases into multi-package repositories. Projects like Babel, React, Angular, Ember, Meteor, Jest, and many others develop all of their packages within a single repository.

Lerna is a tool that optimizes the workflow around managing multi-package repositories with git and npm.

I'm also removing the fixInternalPrivatePackages.sh CI step and just checking in the branched project's package.json as it should be. This will require manual steps if a new branch is added to a dependency but that happens less often than problems from the CI script.

UPDATE Sept 2018:

Lerna is nice but won't help our CI workflow. It's more for dev workstations. We really need a private package repository to keep published versions. QA and feature branches can use latest tags, where production can use our semver-ish tags.

I discovered Verdaccio, and I'm hoping it will resolve our dependency hell.

UPDATE Dec 2018:

The React project itself seems well organized and a good project to imitate. Using yarn workspaces seems like a good idea -- but it would take time to convert everything to a monorepo.

3
  • How'd it go with verdaccio?
    – thisgeek
    Feb 28, 2019 at 21:40
  • @thisgeek -- Couldn't tell ya, I switched jobs :)
    – Nick
    Feb 28, 2019 at 21:52
  • @nick : it seems url is pointing out to some porn website Oct 24, 2022 at 17:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.