10

I'm having difficulty understanding why you should develop internal libraries to be used exclusively for developing internal applications. I appreciate that if I want to use software that someone outside the organisation has written then they can send me their header files and .a or .so files and I can just link it to my project (assuming they are compiled in the same environment).

But why should an internal library be developed just to be linked to an internal application when I have access to the header and implementation files and can just include them in my source tree and compile them all together?

In other words: if some source code is written, how do you decide if it should be compiled into a binary library and linked to your application or just included in the project's source files and compiled regularly?

When I say 'include' files in each project, I do not mean to copy and paste each file into the source tree of the project currently being developed. I mean developing some directory/library (separate to any project) containing common source code that can be included into a project's files in the usual way, i.e. #include.

p.s. I'm talking about c/c++ development here for multiple desktop applications.

25

There are numerous reasons to create libraries & shared libraries, (in .dll or .so files) even for internal usage:

  1. Re-Use across projects is much cleaner
  2. Separation of responsibility - part of your code may be more suitable different developers or teams
  3. You can benefit from improvements in the libraries that other teams make without having to locate the specific code
  4. Faster build times, if the library is stable it doesn't need re-building.
  5. Disk Space - you can have just the library and the headers in the project
  6. If you use shared libraries you can make memory savings with only one copy loaded into RAM even if several programs are using it
  7. You generally end up with better documentation and testing of the libraries
  8. Cleaner design and code - thinking about structuring things into libraries should result in related groups of functionality in each library and you tend to separate generic code, into the libraries, from the application specifics, in the app.
  9. If you have proprietary algorithms in libraries you can restrict access to the source, e.g. not allowing contractors or out sourced teams to get at the source.
  10. Library code may be placed under a different licence to the Application it was originally written as a part of, some companies have even been known to Open Source libraries that they are proud of - gaining major kudos in the Open Source community and sometime major enhancements contributed back.

Some companies even have an accounting practice where projects that create libraries get some reimbursement on each re-use.

  • 1
    Or in a variation of point 9, you could release the library under a different license than main application code. – Michael Borgwardt Aug 25 '17 at 21:05
  • @MichaelBorgwardt - Good point prompting point 10 above. – Steve Barnes Aug 26 '17 at 5:43
  • Also, having some code as a separate library helps to avoid shortcuts while programming, such as "I'll just add this extra parameter here.." and helps to find better ways to implement required features. – Valdas Sep 1 '17 at 13:52
11

Some other possible reasons which may apply to larger, nontrivial projects:

  • Compile times: Huge monolithic C++ projects with thousands of files, thousands of classes, functions, etc. can take a very long time to compile (which hurts productivity if you want to recompile every time you change a few lines of code). Statically-linked and dynamically-linked libraries are compiled independently and do not need to be recompiled if their source hasn't changed.

  • Logical separation of distinct modules or subsystems: Large systems are typically easier to manage if distinct areas of functionality are placed in separate modules, and developers aren't faced with searching through enormous folders/projects containing thousands of files/classes.

  • Boundaries between developers/teams: Developers building separate new functionality at the same time may reduce the potential for merge conflicts if it's possible to have each developer working in different modules.

  • Code which must not be released into a live environment: For example, unit test libraries or 'mock' libraries which are used for developer testing to substitute for a live system component (hardware, APIs, remote systems, databases, etc.)

  • Compiler flags: If you find yourself in the very unfortunate position of integrating with some 3rd party's API which expects a weird compiler flag, then the library can be a "decontamination layer" sitting between the 3rd party API and the rest of the application.

  • Optional features/Optimisation: In large systems, an application might wait before loading certain dynamically linked modules into memory at runtime if they aren't critical to the basic functionality of the application.


In general, many internal projects are often small micro-apps which don't benefit from being broken up into separate libraries. If you're working on a tiny project as a lone developer, then you 'might' not need to worry about splitting your code into libraries (yet...). Don't forget the YAGNI principle.

  • 3
    @downvoter - could you explain the reason for the downvote? Feedback on answers is useful to improve the quality of this site for everybody. – Ben Cottrell Aug 26 '17 at 6:46
4

Your original question might have caused a misunderstanding here for most of those other answers. Since you intend not to copy existing code across projects, but to include the same source files from different projects as references, any "duplicate code" argument becomes pointless, as well as many other arguments presented.

Note this is sometimes (- not always -) a sensible technique. In fact, when you put all the source files you want to reuse across projects into one separate include folder, you already built a libary as well - a source code library, not a binary lib. Especially in C++, when creating generic libraries with templates, it is not unusual to have header-only libs, which just need a simple include, and no separate linking preparations.

So I guess your real question is - when to build source code libs, or when to prefer precompiled binary libs? In this older answer on this site, I discussed some pros and cons of header-only libs, maybe it helps you. The main advantage of source-code libs is, they don't require to be compiled with the same run-time and/or compatible compiler/linker flags as the application which uses them. The drawbacks are the additional compile times, and the requirement to provide access to the source code (which is obviously not a problem for the kind of "internal" projects you have in mind).

  • It's true that I meant a source code library which is common to multiple projects where you just reference the files that you need through the typical #include directive - I didn't know the term for it until you said it and I'll edit the question now. Your linked answer is very helpful too. – Andrew Murtagh Aug 25 '17 at 21:20
  • @AndrewMurtagh: the fact you were so quickly accepting MichaelBorgwardt's answer astonished me, because what he wrote seems to be a either misunderstanding of him or mine. – Doc Brown Aug 25 '17 at 21:22
  • well he did clarify the initial confusion I had about when to group code that would be common to multiple projects into a single package (whether that be in binary libraries or source code libraries) but granted, I was talking about having a single directory of source code that could be shared across projects and not copying and pasting each file into each project as I might need them. – Andrew Murtagh Aug 25 '17 at 21:30
2

I agree with other commenters when they write that you should not duplicate code. In your case, however, it seems that you (or people you work with) are creating libraries for code that is not duplicated elsewhere.

In this case, I caution against premature generalization. There are often times when one feels like a piece of code will be reusable. However, without knowing the intimate details of how the second use case will use such code, it is very easy to spend extra time on "reusability" features that won't actually be useful in the additional cases or to make assumptions that prove wrong in the second case.

Writing a "library" for one use case can turn into a very expensive exercise without any payoff --- I've been bitten by this several times.

Example costs:

  1. Time / energy spent on considering "general" use cases
  2. Time spent making the "library" distributable to your "client" (your own code)
  3. Future pressure to make use of the "library" even if it doesn't fully match the next use case

My general rule is: don't make code into a library unless I have at least 2 separate places where the code is needed.

  • 2
    I have seen almost exactly the same reasons used by some developers as why their code is all in a single >20k line source file. When combined with quirky naming and poor comments you soon have code that it is quicker to rewrite than to maintain. – Steve Barnes Aug 26 '17 at 5:51
  • Great point: I am certainly not arguing against writing maintainable, readable code. Separate files, well-named methods/classes, and package structure are all important components of maintainability. My point is just that the costs of writing + distributing a library are steep to incur when there is only a single use case for the "library." – Sam Aug 29 '17 at 13:37
  • 1
    Conversely I am not arguing that you necessarily need to package and distribute a library in every case, just that writing and structuring as if your code might one day become a library is usually worth a little effort and often pays dividends even if the code never becomes distributed as a library. – Steve Barnes Aug 29 '17 at 14:55
1

why should an internal library be developed just to be linked to an internal application when I have access to the header and implementation files and can just include them in my source tree and compile them all together?

Because if you "just include them in my source tree", you are duplicating code.

The problem with that is that you will not benefit from any improvements (including critical bugfixes) made by the project you copied the code from, nor will they benefit from any improvements you make.

You might think that you can solve this problem by simply copying the newest version of the code into your source tree regularly, maybe even automated by using a submodule in git or something similar. But then you'll constantly have your build break because of incompatible API changes. A library on the other hand has an "official" public API that its developers know cannot be changed without coodinating with clients.

Finally, there might be technical reasons - could keeping part of the code as a library be necessary so it can be loaded optionally or even loaded and unloaded on demand, and thus reduce memory usage when the fuctionality is not needed?

  • 1
    So as I understand, the main reason to make some piece of code into a library is when it is going to be used across different projects (even if they are all internally developed) in order for that code not to be duplicated, updated, maintained separately? And if you are only going to be developing a single project then there is no need to make something into a library (unless you intend on using it in further projects at a later date). – Andrew Murtagh Aug 25 '17 at 20:02
  • @AndrewMurtagh: Yes, that's pretty exactly how I'd put it. Although there may be technical reasons too; I'm not that familiar with C/C++ development - maybe keeping part of the code as a library is necessary so it can be loaded optionally or even loaded and unloaded on demand, and thus reduce memory usage when the fuctionality is not needed? – Michael Borgwardt Aug 25 '17 at 20:09
  • I believe that may be possible through shared libraries. Thanks for clarifying, I'll mark your answer as accepted. – Andrew Murtagh Aug 25 '17 at 20:21
  • Nearly every VCS for the past decade supports external references or submodules. This removed the duplicate code and bugfix problem. Might include more information on why these aren't a viable solution if you still believe they're valid problems. – Sirisian Aug 25 '17 at 20:28
  • Think also in maintenance. 1) The lib can be maintained independently and in parallel. 2) Is code easy to replace. 3) Small code base are easier to manage for small teams and easier to understand for everybody. 4) if time-to-market is critical, you will prefer to have faster builds. Code in libs is code you don't compile over and over in the pipeline (IMO). 5) It's reliable reusable code... You made It ;-) – Laiv Aug 25 '17 at 20:39
1

I would like to elaborate on costs your solution has in the longer run.

Clearly, adding a library to a project has some overhead, all above if it is the first: workflows must be changed, sometimes even infrastructure and some team members may not like it (at first). So the advantages of your solution are obvious, as it imposes less costs now.

However, as your project grows also the costs of the "pseudo-library" will. Assume you have a "pseudo-library" A which is used by an application and an unit tester. Every time you add a cpp to A, you have to add it to both projects, otherwise they will not link.

What if your "pseudo-library" is used by another "pseudo-library" B? You have to add your new cpp in a bunch of projects more. And if B switch to use another library? You will have to delete the cpps from the A in all projects depending only on B.

This all would be for free if a real library would be used. So the question is, how many cpps are needed to justify the move to a real library?

But wait, there is even more collateral: A developer does not like this stupid work of hunting all projects in need of the new cpp down and will add his/her code/classes somewhere into already existing files, which is not a good thing in the long run.

So using the "pseudo-library" can be a first step to break up a monolithic project, but you should not wait too long to make it a real library to be able to use its advantages.

0

But why should an internal library be developed just to be linked to an internal application when I have access to the header and implementation files and can just include them in my source tree and compile them all together?

If the library is only ever used by one application then you probably don't need it as a separate library.

If the library is used by 3,500 applications then you absolutely do need it as a separate library.

What if there's a bug in the library and you need to fix it? Or some statutory or regulatory change comes along that means you have to change the way the library works?

If it's in a separate library then you can (potentially) fix the library, re-test it and re-deploy it and every application benefits from the fix.

If it's only in source code that's "local" to each application, then you have to change, re-build, re-test and re-deploy every application individually. That's a much bigger (i.e. more costly) exercise.

protected by gnat Aug 31 '17 at 9:09

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.