Why develop internal libraries for internal applications?

Question

I'm having difficulty understanding why you should develop internal libraries to be used exclusively for developing internal applications. I appreciate that if I want to use software that someone outside the organisation has written then they can send me their header files and .a or .so files and I can just link it to my project (assuming they are compiled in the same environment).

But why should an internal library be developed just to be linked to an internal application when I have access to the header and implementation files and can just include them in my source tree and compile them all together?

In other words: if some source code is written, how do you decide if it should be compiled into a binary library and linked to your application or just included in the project's source files and compiled regularly?

When I say 'include' files in each project, I do not mean to copy and paste each file into the source tree of the project currently being developed. I mean developing some directory/library (separate to any project) containing common source code that can be included into a project's files in the usual way, i.e. #include.

p.s. I'm talking about c/c++ development here for multiple desktop applications.

Answer 1

There are numerous reasons to create libraries & shared libraries, (in .dll or .so files) even for internal usage:

1. Re-Use across projects is much cleaner
2. Separation of responsibility - part of your code may be more suitable different developers or teams
3. You can benefit from improvements in the libraries that other teams make without having to locate the specific code
4. Faster build times, if the library is stable it doesn't need re-building.
5. Disk Space - you can have just the library and the headers in the project
6. If you use shared libraries you can make memory savings with only one copy loaded into RAM even if several programs are using it
7. You generally end up with better documentation and testing of the libraries
8. Cleaner design and code - thinking about structuring things into libraries should result in related groups of functionality in each library and you tend to separate generic code, into the libraries, from the application specifics, in the app.
9. If you have proprietary algorithms in libraries you can restrict access to the source, e.g. not allowing contractors or out sourced teams to get at the source.
10. Library code may be placed under a different licence to the Application it was originally written as a part of, some companies have even been known to Open Source libraries that they are proud of - gaining major kudos in the Open Source community and sometime major enhancements contributed back.

Some companies even have an accounting practice where projects that create libraries get some reimbursement on each re-use.

Answer 2

Some other possible reasons which may apply to larger, nontrivial projects:

• Compile times: Huge monolithic C++ projects with thousands of files, thousands of classes, functions, etc. can take a very long time to compile (which hurts productivity if you want to recompile every time you change a few lines of code). Statically-linked and dynamically-linked libraries are compiled independently and do not need to be recompiled if their source hasn't changed.

• Logical separation of distinct modules or subsystems: Large systems are typically easier to manage if distinct areas of functionality are placed in separate modules, and developers aren't faced with searching through enormous folders/projects containing thousands of files/classes.

• Boundaries between developers/teams: Developers building separate new functionality at the same time may reduce the potential for merge conflicts if it's possible to have each developer working in different modules.

• Code which must not be released into a live environment: For example, unit test libraries or 'mock' libraries which are used for developer testing to substitute for a live system component (hardware, APIs, remote systems, databases, etc.)

• Compiler flags: If you find yourself in the very unfortunate position of integrating with some 3rd party's API which expects a weird compiler flag, then the library can be a "decontamination layer" sitting between the 3rd party API and the rest of the application.

• Optional features/Optimisation: In large systems, an application might wait before loading certain dynamically linked modules into memory at runtime if they aren't critical to the basic functionality of the application.

---

In general, many internal projects are often small micro-apps which don't benefit from being broken up into separate libraries. If you're working on a tiny project as a lone developer, then you 'might' not need to worry about splitting your code into libraries (yet...). Don't forget the YAGNI principle.

Answer 3

Your original question might have caused a misunderstanding here for most of those other answers. Since you intend not to copy existing code across projects, but to include the same source files from different projects as references, any "duplicate code" argument becomes pointless, as well as many other arguments presented.

Note this is sometimes (- not always -) a sensible technique. In fact, when you put all the source files you want to reuse across projects into one separate include folder, you already built a libary as well - a source code library, not a binary lib. Especially in C++, when creating generic libraries with templates, it is not unusual to have header-only libs, which just need a simple include, and no separate linking preparations.

So I guess your real question is - when to build source code libs, or when to prefer precompiled binary libs? In this older answer on this site, I discussed some pros and cons of header-only libs, maybe it helps you. The main advantage of source-code libs is, they don't require to be compiled with the same run-time and/or compatible compiler/linker flags as the application which uses them. The drawbacks are the additional compile times, and the requirement to provide access to the source code (which is obviously not a problem for the kind of "internal" projects you have in mind).

Answer 4

I agree with other commenters when they write that you should not duplicate code. In your case, however, it seems that you (or people you work with) are creating libraries for code that is not duplicated elsewhere.

In this case, I caution against premature generalization. There are often times when one feels like a piece of code will be reusable. However, without knowing the intimate details of how the second use case will use such code, it is very easy to spend extra time on "reusability" features that won't actually be useful in the additional cases or to make assumptions that prove wrong in the second case.

Writing a "library" for one use case can turn into a very expensive exercise without any payoff --- I've been bitten by this several times.

Example costs:

1. Time / energy spent on considering "general" use cases
2. Time spent making the "library" distributable to your "client" (your own code)
3. Future pressure to make use of the "library" even if it doesn't fully match the next use case

My general rule is: don't make code into a library unless I have at least 2 separate places where the code is needed.

Answer 5

why should an internal library be developed just to be linked to an internal application when I have access to the header and implementation files and can just include them in my source tree and compile them all together?

Because if you "just include them in my source tree", you are duplicating code.

The problem with that is that you will not benefit from any improvements (including critical bugfixes) made by the project you copied the code from, nor will they benefit from any improvements you make.

You might think that you can solve this problem by simply copying the newest version of the code into your source tree regularly, maybe even automated by using a submodule in git or something similar. But then you'll constantly have your build break because of incompatible API changes. A library on the other hand has an "official" public API that its developers know cannot be changed without coodinating with clients.

Finally, there might be technical reasons - could keeping part of the code as a library be necessary so it can be loaded optionally or even loaded and unloaded on demand, and thus reduce memory usage when the fuctionality is not needed?

Answer 6

I would like to elaborate on costs your solution has in the longer run.

Clearly, adding a library to a project has some overhead, all above if it is the first: workflows must be changed, sometimes even infrastructure and some team members may not like it (at first). So the advantages of your solution are obvious, as it imposes less costs now.

However, as your project grows also the costs of the "pseudo-library" will. Assume you have a "pseudo-library" A which is used by an application and an unit tester. Every time you add a cpp to A, you have to add it to both projects, otherwise they will not link.

What if your "pseudo-library" is used by another "pseudo-library" B? You have to add your new cpp in a bunch of projects more. And if B switch to use another library? You will have to delete the cpps from the A in all projects depending only on B.

This all would be for free if a real library would be used. So the question is, how many cpps are needed to justify the move to a real library?

But wait, there is even more collateral: A developer does not like this stupid work of hunting all projects in need of the new cpp down and will add his/her code/classes somewhere into already existing files, which is not a good thing in the long run.

So using the "pseudo-library" can be a first step to break up a monolithic project, but you should not wait too long to make it a real library to be able to use its advantages.

Answer 7

But why should an internal library be developed just to be linked to an internal application when I have access to the header and implementation files and can just include them in my source tree and compile them all together?

If the library is only ever used by one application then you probably don't need it as a separate library.

If the library is used by 3,500 applications then you absolutely do need it as a separate library.

What if there's a bug in the library and you need to fix it? Or some statutory or regulatory change comes along that means you have to change the way the library works?

If it's in a separate library then you can (potentially) fix the library, re-test it and re-deploy it and every application benefits from the fix.

If it's only in source code that's "local" to each application, then you have to change, re-build, re-test and re-deploy every application individually. That's a much bigger (i.e. more costly) exercise.