2

TL;DR I want to change the way our current enterprise code base is constructed to utilize NuGet packages for the inter-project references, but that seems to present some challenges when considering how that affects the team development environment. Please see the various sections below in bold for the details.

Forgive me if this sounds like a bit of a "noob" question. I'm actually a senior C# developer, but I've been working on the same project for about the past 22 years. Many of the things we're doing today are simply done this way because we're working off of an existing framework, code base, and development model. Lately however, we have begun to undertake the task of modernizing the system. Some of this involves iterative migrations from things like XML web services / .Net Remoting to Web API. One big part of this change that I am wanting to promote is the use of NuGet packages to handle inter-project dependencies. That means a transition from the way it's been done, and introduces some questions as to what should be done to promote best practices for development.

What we're currently doing We have an enterprise application (several, actually) broken up into the logical projects that represent the various libraries for the data access components, business logic, and UI / service layers of the application. Consider libraries like

* com.example.retail.dataacess
* com.example.retail.business
* com.example.business
* com.example.ServiceLayer
* com.example.WebApp
* com.example.nightly

Without getting all off into the weeds about what these hypothetical libraries would do (I am sure you get the gist), What would currently happen is that if you made a change in com.example.dataaccess, you would unit test that change and check it into source control. The automated build process would then build the project and the dependency graph(s) of it and other projects in the code base, and then all the resulting DLLs and EXEs get deployed to a common BIN folder. You then run a script that downloads that BIN folder to your local machine, where you can then make changes to other libraries (like com.example.business that depend on the changes you just made, and so on up the dependency tree.

The project files you work with have dependencies on the various libraries, which are referenced assemblies all in a common BIN folder that's set up in Visual Studio as a common search location for dependencies. Of course if you want to integration test something before checking it in, you can just copy the DLLs over from the project output director to that common BIN folder and then go work on the dependent projects, and not check anything in until all levels of your testing are complete. It's a little arduous, but you get used to it.

What I want to do going forward I've worked in another organization where they handled their dependencies through NuGet packages, which is obviously the more contemporary way to handle this problem. I like it because it does allow you to make changes to a library's code, but only apply them to projects that need them by having those projects reference the newer version of the NuGet package. If a project doesn't need to know about the change you made, then no need to update it. That does present a potential problem down the road though, that I want to address as well. However once I made a change in a dependent library, because the dependencies are now in NuGet packages and not just DLLs, I can't simply copy DLLs around and test other projects in isolation, unless of course I change the project files to use direct file dependencies (and then don't forget to revert them back). Ideally, we would do something similar to what we do now (i.e. build a -beta NuGet package locally and reference it during testing), and then check the whole graph in once the change was complete. But how would you go about doing that without having to do all kinds of gymnastics with the project files, as far as changing the project references and / or repo sources? How is this typically done in an enterprise environment where teams are working on changes and multiple people might be working on different aspects of the same changes, albeit in different projects in the dependency graph?

Additional questions and concerns As I mentioned before, one potential pitfall I see to my idea of using NuGet packages is that if you introduce a change in a library, but some other library doesn't need that changes (and thus continues to reference v1.1 of the package, while your changes go into v1.2), if someone else comes along next release and adds a change to that library that the aforementioned project does need visibility to (say, v1.3), then that project will implicitly inherit my v1.2 changes. While my changes were presumably well-tested, you can predict the unpredictable. If my v1.2 change now breaks something in the dependent project, it will only appear as the result of upgrading the reference to v1.3, which might complicate the RCA when trying to fix the defect. This kind of stuff didn't happen (as much) in our current model because every change went into every project in the Release Candidate and, assuming that the system testing was comprehensive before the Release, those kinds of bugs would show up, and you'd find out what you broke and never thought to test. (or didn't know the dependency exists, as is often the case in a complex system that's 20+ years old and counting.)

Correct me if I'm wrong, but it's my understanding that once you add a new version of a NuGet package to a repo, you can't remove it. You can update it, and (maybe even?) deprecate it. So, I want to avoid conditions where we pollute the repo with bad versions of packages that have bugs. Maybe using the beta tag until the RC solves this problem?

Any thoughts about how you're doing it in an enterprise environment are greatly appreciated!

4
  • "which is obviously the more contemporary way to handle this problem" is never a good argument for changing you process - you need to have a real benefit from this, "being contemporary" does not count. You mentioned "it does allow you to make changes to a library's code, but only apply them to projects that need them by having those projects reference the newer version of the NuGet package." - but is it really what you want? ...
    – Doc Brown
    Commented Oct 17, 2023 at 6:11
  • 1
    In a system consisting of many small, individual projects or micro services, that is ok. In a large monolith, you typically want to avoid mixing different versions of the same library. So are you sure your system really requires different versions of the same lib?
    – Doc Brown
    Commented Oct 17, 2023 at 6:13
  • Use the right tool for the job. Project references exist for a reason. They are much easier and cleaner to use. Nugets on the other hand are a nightmare to use in development loops. They are meant for slowly changing libraries. Versioning is a huge nightmare as well, even when using nugets the way they are designed to be used. I haven't seen a single company to successfully work this. Even Microsoft is having a hard time syncing their nuget interdependencies. Avoid it if you can.
    – Ccm
    Commented Oct 17, 2023 at 21:40
  • Does raise the question: at the end of the day, is there a single monolithic result artifact from all this, or are the libraries consumed by lots of microservices? In some senses, a nuget package is a microservice ..
    – pjc50
    Commented Oct 18, 2023 at 15:18

1 Answer 1

4

Tangent: your project structure

First of all, a massive elephant that is proving to be a bit of a distraction / cause for general concern: if I understand your explanation correctly, you have separated your codebases by layer, i.e. instead of cutting up a monolith vertically like microservices do (by bounded context), you've cut it up horizontally (by individual layers of those services).

I don't like that approach. I also think it separate some layers further from each other, leading to a significant increase in overhead cost in terms of handling inter-layer dependencies and references. This will inevitably impact how much effort is involved in managing anything related to this (and this question falls squarely into that category).

However, the core question you're asking is the same even if you were to assume a more common microservice architecture where you have separated services but not separated layers. So I'm going to answer this question in a way that sidesteps your project organization because it's not the core focus here.


How to Nuget

Ideally, we would do something similar to what we do now (i.e. build a -beta NuGet package locally and reference it during testing), and then check the whole graph in once the change was complete. But how would you go about doing that without having to do all kinds of gymnastics with the project files, as far as changing the project references and / or repo sources?

I'm going to establish a few points that are necessary to get to what I want to focus on. I'm skipping the justifications because the answer would explode in size.

  • Nuget packages are intentional releases of versions. They indicate an intentional action on the part of the package developer.
  • Because the consumer can consume Nuget package versions at their own leisurely pace (and update references to newer versions as they see fit, or not), it is paramount to keep older versions of your package available, ideally without ever expiring.
  • Semver is the established standard for managing an evolving version that respects older versions and how consumers can choose the right version for their use case.
  • For the purposes of testing, it would be bad form to have to change how you consume your dependency, because then you're not testing the real thing anymore. This argument is no different (in spirit) from having different methods for a "real" call and a "test" call in a class. Any difference between the two is likely going to obfuscate what should have been caught as a test failure. Ideally, you want your tests (even for non-final Nuget packages) to use the real Nuget dependency.

There are two points I want to make here. The first is a bit of an idealistic principled point, but it is the overall better approach.

1

TL;DR: A Nuget package denotes the boundary between two components that should really have a separate lifecycle. In doing so, you significantly mitigate the need for experimental package release in the first place.

Nuget packages are a non-trivial process. You can't feasibly build and release a package for every tiny change you make to your code, plus it would take a non-zero amount of time to get all these changes build and onto your feed. Can it be done? Sure. Will it be nice and quick for a developer who is mid-development and in "the zone"? Absolutely not

What this means is that Nuget packages act as a slow gate, and you should really be designing your process in a way that minimizes the amount of times you need to release such a Nuget package.

Speaking in general microservice terms (which I know isn't your specific use case but humor me here), you should try and finalize your changes to Service A before you even begin implementing how Service B consumes the new version of A.
By minimizing the back-and-forth between developing A and B, you minimize how often you have to cross the slow bridge of Nuget package releasing.

In an ideal world, I would suggest you hold to the standard of testing A in isolation and fully confirming it follows the standards before you even begin to implement in B. If you do so, you actually don't need to ever use an experimental Nuget package from A in service B, because you will have vetted A before it ever needed to be a Nuget package.

This is an idealistic point, and not necessarily a fully realistic one. However, the other side of the spectrum, where you actively choose to develop your features across multiple independent services (including Nuget bridges between them) is going to be a really inefficient process with significant complexity both in scheduling their release and avoiding/handling regressions, on top of requiring developers to actively manage several services at the same time.

All of these things increase the cognitive load on your developers and it will significantly complicate your development process. Try to respect the individual nature of your individual projects. A Nuget package denotes the boundary between two components that should really have a separate lifecycle.

2

TL;DR: Semver pre-release versioning solves the problem for you.

If you accept these bullet points, and you don't fully adhere to the (admittedly idealistic) previous point, then the remaining question is "how can I pre-release Nuget packages that are not a final version yet?". The answer to that will be the answer to your question.

"Pre-release" is the operative word here. Semver actually account for the need to have alpha and beta packages, by using an established version number format that indicates the pre-release nature of it. Emphasis mine (link):

A pre-release version MAY be denoted by appending a dash and a series of dot separated identifiers immediately following the patch version. Identifiers MUST be comprised of only ASCII alphanumerics and dash [0-9A-Za-z-]. Pre-release versions satisfy but have a lower precedence than the associated normal version.

In other words, the following list of versions is considered to be in order of version number:

    1.0.0
    1.0.1
    1.0.2
    1.1.0
    2.0.0-alpha
    2.0.0-beta
    2.0.0
    2.0.1

Small detail: alpha and beta are sorted by standard alphabetical order. Make sure that if you have multiple pre-releases of the same version, that you label them accordingly. The name you pick is completely up to you.
If you still insist on having many experimental releases for the same version, make sure to adequately prefix your numbering to ensure alphabetical sorting is maintained, e.g. 2.0.0-rc001, 2.0.0-rc002, ..., 2.0.0-rc099, 2.0.0-rc100, 2.0.0-rc101, ...

What this means is that if a consumer is set to consume the 2.0.0 package, and we're living in a world where 2.0.0-alpha is the most recent release (according to the above chart), then that consumer will take the 2.0.0-alpha version because it falls under the umbrella of 2.0.0.

However, without needing to change the consumer, if you then release the actual package 2.0.0, your consumer will automatically take the 2.0.0 (newer, non-experimental) version instead of the 2.0.0-alpha that it used to work with (of course, this requires the consumer to rebuild, but that's inherently how Nuget is designed to work).

What this means is that you can set your consuming service to already "sign up" for the new version, and the way semver sorts its versions will ensure that it will take the best version available (i.e. experimental only if there is no final version), without you needing to constantly manually update precisely which pre-release version it should consume.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.