4

My company develops many relatively small projects that a lot of times do the same things and have a similar structure. (e.g. read/write to a databse, data pre processing, building a query etc.). This creates a natural urge to create libraries and frameworks that either implement some of the repetitive logic and/or force a certain uniform program structure.

These libraries are often developed by developers without a large enough sample size of projects to be generic and are later not maintained properly because the team that developed them has different priorities than to improve their framework, especially for use cases of other teams.

Working with these libraries has caused me to automatically be repulsed by the idea of in house libraries and frameworks, and I rather copy the code from project to project, which will make it a hell lot easier to change than having to deal with releasing a new library version, even though I hate inconsistency and code duplication.

I would like advice on how to properly deal with this situation.

Improve this question
11
  • 1
    What's your role in the company? The most likely answer is quite likely "you personally can't".
    – Philip Kendall
    Commented Aug 13, 2022 at 17:15
  • 3
    It sounds to me like your problem is in dependency management, rather than technical debt: if copying and pasting the solutions allows you to make them fit for your purpose, why are you not able to change a central copy of them with the same aim?
    – IMSoP
    Commented Aug 13, 2022 at 18:00
  • 2
    @krezno Right, which is why I said that it's a problem of dependency management (and, it sounds like, team co-ordination and management). It was a suggestion to reframe the problem, rather than a solution, which is why I didn't post it as an answer.
    – IMSoP
    Commented Aug 13, 2022 at 18:09
  • 2
    It's a sad fact that most "frameworks" and "libraries" rarely scale beyond the individual developer who made them, and often don't generalise beyond the existing sample of projects they were written to serve. However once you have a dozen copies of code, and if you're expected to maintain all dozen copies, that too can become difficult to maintain due to a proliferation of subtle variations, or the need for every framework change to be propagated manually across many copies. I've come to the conclusion that this is often symptomatic of people doing too much with too little experience.
    – Steve
    Commented Aug 13, 2022 at 18:18
  • 3
    Ok, I found this question from 2017, where the answerers agree to your observations, but don't give a receipt for solving the issues you mentioned
    – Doc Brown
    Commented Aug 13, 2022 at 19:17

5 Answers 5

Reset to default
5

There is no general solution to this problem. Welcome to approximately every organization that develops software. In many ways this is precisely the problem that the open source community has, as well.

Some person writes something useful. A few other people use it, too. Then a few more. Soon bugs are discovered and enhancements are requested, but the original maintainer is off doing something else. Copy and paste was your only recourse, until GitHub came along.

IMSoP definitely alludes to something important in their comment on your question. The obvious problem is managing dependencies, but this is only part of the solution. The organization needs a large amount of automation as well, otherwise distributing and testing these libraries becomes prohibitively labor-intensive.

At a high level, these libraries need:

  • Excellent automated test coverage.
  • An automated build system.
  • A system to run automated tests.
  • A system that empowers collaboration, so the original maintainers can retain veto power over changes, but are not solely responsible for implementing those changes (think: code reviews or pull requests).
  • A centralized package management system.
  • The ability to push new versions of packages to the package repository.
  • To use Semantic Versioning, which allows you to communicate bug fixes, versus feature enhancements, versus breaking changes using version numbers.
  • A hosting solution for developer documentation that integrates with existing continuous integration/continuous delivery pipelines (CI/CD), so documentation changes can be deployed automatically.
  • A good distributed version control system that allows people to fork code repositories, and collaborate on writing code.
    • This allows for customizations that the original maintainers reject, but your project finds useful.
  • Some tool that allows the organization to identify and change the maintainers of these libraries.

And all of this will likely need to exist in a walled garden, of sorts, that is only available for authorized developers with your organization. Perhaps even within your organization's internal network.

You basically need to replicate the open source community inside your organization (along with similar collaboration tools) — an internal developer ecosystem. Implementing this is no small feat.

If your organization already uses a tool like GitHub, GitLab, Jira, Azure DevOps, etc, look at the features and components it supports. You might already have all the tools necessary to implement this, and your organization only needs a champion for the cause.

After this is all put together, maintainers spend time reviewing bug fixes and feature enhancements instead of writing them. You spend time fixing bugs and adding enhancements to the libraries as part of your regularly scheduled work (you just work in multiple code repositories), so management might not even care that you are spending time doing that. The CI pipeline performs all necessary testing. The CD pipeline packages things up and pushes them to the central package repository (including documentation changes). And when maintainers do not want to incorporate your change, you can fork their repository and implement the change yourself, and create your own package. Breaking changes can be introduced more safely when using a proper package manager and Semantic Versioning.

Improve this answer
2
  • 2
    (off-topic, but related) And while we're at it, I'd like a pony, too!
    – Greg Burghardt
    Commented Aug 13, 2022 at 21:46
  • original maintainer is off doing something else. don't even need to be off. It's as simple as fixing the lib is out of the scope of any ongoing project. Basically nobody wants to pay the cost of fixing bugs, update the dependency in all the projects implementing the lib and manage a new deployment and test in every single env up to production. When it comes to technical debt, everybody pretendes not be aware of it and ignore it even when it's bleeding the budget for maintenance. Paying back tech debt is a lost war from the very first day.
    – Laiv
    Commented Aug 16, 2022 at 9:47
2

Easy: make someone owner of that library and assign serious time to let him/her take care of it.

If you do scrum however there may be a problem... This is a good example of where scrum fails miserably. No product owner will put an issue that deals with this sort of maintenance in the sprint. It takes a strong team to incorporate this kind of work into their routine (it is basically the team's responsibility, they need to take this up with any issue that depends on it and estimate accordingly).

Improve this answer
6
  • 1
    Yes we work in scrum. I think maybe the somewhat tolerable sweetspot is limiting libraries to your team and not guaranteeing backwards compatiblity to anybody else that uses it. That way you can somewhat contain the uses to the people that sit in the same room as you.
    – krezno
    Commented Aug 13, 2022 at 19:45
  • 1
    I have worked with a product owner who frequently allowed the technical team to add maintenance tasks to the backlog, so they could be included in upcoming sprints. This included all kinds of technical debt reduction, and also refactoring code into libraries if it was likely to be used outside the project, or working on those libraries. But I agree, product owners like that particular one are a rare breed.
    – Dawood ibn Kareem
    Commented Aug 14, 2022 at 2:38
  • 2
    The issue with scrum that you point out is that technical details should not be put before a product owner. It doesn't matter whether there is a library or not. The PO needs to be told how long it will take to develop their feature / user story, not exactly what the change consists of. Similarly, if it takes 4 days to develop something and 1 day to write tests for it, then you tell the PO that development takes 5 days. Don't let them make a choice where the easy out is detrimental to someone else. That's just asking for friction between the PO and devs.
    – Flater
    Commented Aug 14, 2022 at 7:22
  • @Flater: your comment is fine, but if I think it takes 4 days for developing something and 1 day to write tests, I add another 3 days for debugging and retesting (and some extra time for documentation, and some buffer for more unforeseen things). That is IMHO the frequent error when giving estimates - underestimating the time for the necessary extra activities .
    – Doc Brown
    Commented Aug 15, 2022 at 13:45
  • @DocBrown I don't disagree with your point but it wasn't the focus of the message, and that kind of padding would generally already be included in the example amount of 4 days.
    – Flater
    Commented Aug 15, 2022 at 14:15
2

Though I agree with @GregBurghardt's answer in general that for library development in larger organizations one has to scale the organizational infrastructure with all the measures he mentioned, to my experience, there is a middle ground for smaller teams, which does not require to "replicate the open source community inside ones organization", at least not in full . In case

  • the group of people accessing a common library is small (5 to 6 persons at most)

  • you have shared code ownership for the team (so anyone can fix bugs on the libs)

  • you have tools for reverse-lookup where in the whole code base a certain library and/or library function is used (so anyone who starts to change something in a lib can make an impact analysis beforehand)

then a certain amount of library/framework development is possible and manageable without causing too much hassle. Of course, you have to bring up some discipline to follow a few strategic and tactical rules:

  • "Small is beautiful"! Don't put things into common libraries "just in case" - only put code there which was used at least two or three times somewhere.

  • Avoid developing large "frameworks", which too easily get interlocked with the applications depending on them. They often suffer from exactly the issues you mentioned in your queestion.

  • Ideally, put only stuff into a lib where the API can be kept highly backwards compatible, and avoid incompatible changes to function signatures or function semantics

  • If there is code which brings in 3rd party dependencies, make sure will be kept in separate libraries, so noone will be forced to introduce a dependency they don't need.

  • More general: don't be tempted to throw too much stuff into a "commonlibrary" dumpster. Keep your libs clearly separated by topics and responsibilities.

Some of the points on Greg's list most teams should already have in place today, regardless whether they develop common libs or not, so it is not necessarily extra effort. The most obvious thing is state-of-the art version control. Moreveover, automated builds and a CI system are pretty standard today, to make sure noone accidentally breaks the build (and in case this happens, each team member should be able to fix that, regardless if the bug is in a lib or elsewhere). Automated tests are also helpful, but usually it is sufficient to use the tests your team has already written for the software based on the common libs, so this also might not cause too much extra effort.

However, the larger the number of people working with these inhouse libs gets, the more management and communication effort will be needed. At a point where you split one team into two, to allow them more individual development, you have to decide whether each of the teams will have their own "common library", or if the extended communication and maintenance effort is still balanced by the time and effort saved through reusage.

Improve this answer
1

If the library in question is not yet developed for your use case, it is causing you more work than expanding the library would cost, and developers have "other priorities" which prevent them from expanding the library, then this is a priority problem.

You already know the solution, the company just doesn't allot you the time needed to fix it.

To be clear here, if adapting the library takes more time and only applies to your use case and there are no expected future reusability gains, then you should not be foisting it into the library.

Talk to your manager and explain the amount of delays encountered by using the library as is. Either provide an estimate on how long it would take to expand the library, or get your manager to ask the team that develops the library for one (provide a detailed analysis of the change that needs to be made).

If you are correct and management listens to reason, you'll get your change. If you're wrong, then this is not an issue that should be addressed. If they won't listen to reason, then the company chooses to perpetuate the issue and you can't change that.

Improve this answer
2
  • 1
    This's the only reasonable answer to the problem. It's not a technical issue that can be solved with sophistication. It's a business problem that needs to be approached as part of the company's business strategy. It's something to encourage or enforce by the CTO, the only who can allow project managers to give the proper priority to this problem. If they don't understands the value of paying back the technical debt, don't expect managers and developers to do so for free. They will be left alone in this fight.
    – Laiv
    Commented Aug 16, 2022 at 10:05
  • 1
    @Laiv Well put. You've also reminded me to point out that it may very well be that OP may be running into inefficiency due to a lack of library expansion, but that the company may accept this to be the case, e.g. because the library developers are off working on something else that renders more value to the business than expanding the library would render. In such a case, it's not wrong to have these people not expand the library, it's a matter of priorities, which neither OP nor the library developers have a say over.
    – Flater
    Commented Aug 16, 2022 at 10:08
1

The worst kind of technical debt is having multiple copies of a library, each slightly different in incompatible ways.

So your organisation should make that absolutely forbidden. If anyone needs added behaviour, or changed behaviour, the original creator or maintainer of the library should make the changes, either in a way that is backward compatible or configurable at build time or at run time. (Not at build time obviously if you distribute binaries, unless you can distribute different binaries).

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.