Why don't software libraries solve all our problems? [closed]

Question

Modular programming and reusable software routines have been around since the early 1960's, if not earlier. They exist in every programming language. Conceptually, a software library is a list of programs, each with its own interface (entry points, exit points, function signatures) and state (if applicable). Libraries can be high quality because modules are focused on solving narrow problems, have well-defined interfaces, and the cost can be amortized across all future software programs that will ever use them. Libraries are purely additive: adding new modules does not introduce bugs or limitations in existing modules.

So why don't software libraries solve all our problems? Why can't we write software as merely a composition of high-quality software modules?

Can we fix these problems with software libraries and unleash the full potential of this incredibly powerful mechanism for writing high-quality programs faster? Or are these problems intrinsic to libraries and can never be solved?

Note: Several comments said that I have too many questions. Please treat the above questions as the real questions and everything that follows as points for discussion.

It is true that software libraries are widely used. Some programming languages such as C, Java and Python have enormous software libraries. But there are numerous problems.

1. Some well-known languages have less than ideal library support (e.g. C++, Lisp). To some extent this is mitigated by piggybacking on a virtual machine platform (e.g. JVM, CLR). A corollary question, should all future software be written for a virtual machine platform to increase library support? This is problematic for scripts that don't want to incur the cost of launching a virtual machine every time.
2. There is a lot of "reinventing the wheel". Have you ever written a linked-list module in C? Yes, of course you have. I don't enjoy writing linked lists in C, but what is the alternative?
3. Can a given library (e.g. libfoo-0.1.2) be trusted as as a basis to write your important software? Is the library tested, documented, and does it implement the features you need? How can you tell?
4. Learning a library's API can be as time consuming as learning a whole new programming language.
5. If a bug is discovered in a library, what is the proper procedure for fixing the bug in your software? How should the bugfix be distributed to all library users?
6. How should libraries be built and distributed? (For example, autotools obviously got it wrong.) Semantic Versioning is good, but how can this be verified and enforced?
7. How should library dependencies be handled? Can we download and install automatically? Will the versions and licenses be compatible?
8. Is the license for a given library compatible with your software? How can you tell?
9. What should you do if the library is abandoned before starting your software? What about after starting your software?

Answer 1

Libraries are great and certainly solve a lot of problems very quickly. However they never solve every problem and even gluing together many libraries isn't enough to solve many specific use cases.

Programmers are not typists. We've had that discussion before. Despite all the libraries, frameworks and out of the box solutions, there is always something your framework, library or SaaS product is not going to do for you. You're going to need to store your data in a different way, you're going to need to read your data faster, you're going to need to validate input in a way your libraries don't.

In any programming project of significant size, there is going to be some problem you need to solve that's so specific to your use case, so unusual, or so complex/simple that no library covers it properly. You're going to need to write your own code, your own objects or maybe even your own libraries or frameworks.

There's also lots of business reasons libraries aren't or can't be used. Maybe we can't use this library because of its license. Maybe we don't trust the source, or we'd have to modify it, and we don't trust an altered library. Maybe management believes all code needs to be purely owned, created and maintained by the company. But for various reasons, good, bad and spurious, many IT shops don't or can't use as many libraries as possible.

Some of these problems simply can't be fixed; some management types (or coder types!) are simply never going to trust code they don't own. The quality of libraries will never trump dogma.

Other problems are more solvable (quality of libraries, licensing) but the fact of the matter is that the legal and practical implications are often large and complex. Until all libraries are public domain and are 100% stable while containing all possible features (with perfect performance!) there's always going to be a reason to write your own code rather than use a library.

This isn't to say that libraries don't result in better programs; libraries solve a lot of problems and keep you from endlessly solving and debugging the same problems or working on hacked solutions. But the practical barriers mean they'll never be a panacea.

Answer 2

Libraries were created by someone to fit some purpose they had. It might sound like a problem you want to use the library to solve, but it might not fit your exact usage. When considering a library you do have to evaluate it.

As a developer, one of your responsibilities is to think about how any library may fit into your software, and whether or not you can make your software and libraries work well together. Certainly as you gain more experience that becomes easier, but when you're a blank slate on libraries, there is a higher initial ramp-up time.

That said, I did find your questions interesting so I took my shot at answering them.

Some well-known languages have very poor library support (e.g. C++, Lisp). To some extent this is mitigated by piggybacking on a virtual machine platform (e.g. JVM, CLR). A corollary question, should all future software be written for a virtual machine platform to increase library support? This is problematic for scripts that don't want to incur the cost of launching a virtual machine every time.

Define "Very poor." C++ has boost, qt, Windows MFC and Win32 API, poco, WxWidgets, and those are just the general use frameworks and libraries similar to what you'd get with, say, the Java API. There are C++ libraries for many things, without having to factor in a CLR or JVM. Plus you can take advantage of the many C libraries out there (glib, curl, Apache Portable Runtime) and wrap them in C++ if you want to feel more C++-ey.

Lisp, though. There are many varieties of Lisp. You have modern Lisps like Clojure (on the CLR and JVM) and they can take advantage of their respective platforms. Common Lisp as a standard has many implementations, and I know there are companies out there building software in Common Lisp. I don't know enough about the kind of library support it has though.

Also, whether or not you should use a VM-based language. That's really up to you and what you want to use and what your company/employer/team are fine with. I will use what I think is a suitable tool for the job, as well as what I'm productive in for a given task.

I use Python and Ruby heavily because it's easy for other people to understand and I can quickly do a task in them.

There is a lot of "reinventing the wheel". Have you ever written a linked-list module in C? Yes, of course you have. I don't enjoy writing linked lists in C, but what is the alternative?

I have, but only as an academic exercise. There are several widely used libraries that provide data structures like linked lists for C, namely Apache Portable Runtime and glib.

The general alternative is to not leap before looking - you'll find reinvention in all languages, and much of that may be due to lack of knowledge about existing solutions.

Also, languages and libraries evolve over time. You're not going to find a library that fully suits your needs at some point in time, and in certain fields, algorithmic invention might be very commonplace.

Can a given library (e.g. libfoo-0.1.2) be trusted as as a basis to write your important software? Is the library tested, documented, and does it implement the features you need? How can you tell?

I base it on usage, community, maintenance activity, and if I must, I will check out its bug/task queue and see if it moves along smoothly. If it's commercially supported, I like to see if they actively update it. I also go by recommendations from developers I trust, as well as from communities like StackOverflow, where certain libraries are just mentioned significantly more than others.

There are libraries which I will trust without question, like boost and qt, and I will trust things from Google to perform correctly because I know Google eats their own food and uses their own code internally to process the bajillions of gigabytes of data they get every day.

For other things, I will evaluate it, read the code, and tailor it or scrap it if it's unsuitable.

Learning a library's API can be as time consuming as learning a whole new programming language.

Languages are easy to pick up, especially once you have experience in other languages and a similar paradigm. Ultimately they boil down to you expressing your logic in text, and being sure the compiler/interpreter knows what you mean.

This could be a factor of poor documentation, or you're using a framework that is significantly scoped (like the Java API). But the secret is, you don't necessarily have to learn 100% of a library to be effective using it. I consider myself a proficient Java developer, but I have not touched parts of the Java API because I have never had a need to.

Single purpose libraries are easier to learn in that regard - they have more limited functionality and hopefully documentation is simpler for it.

If a bug is discovered in a library, what is the proper procedure for fixing the bug in your software? How should the bugfix be distributed to all library users?

I have a lot of experience with this (making boost work on Solaris, making boost work on AIX, making tons of other things work on Solaris, like graphics libraries). Fixing it is the easy part.

If this is proprietary code, then my company would do something like patch releases if it was minor issues. Basically a bunch of fixes rolled into one, and then we notified customers via email that we had a new patch release.

For critical or per-customer issues, the fixes were put into a quick patch and we would either distribute just patched files to users, or release a quick patched release.

For open source projects - file a ticket with the project's bug tracker, and if you fixed it, submit your fix recommendation and a patch.

How should libraries be built and distributed? (For example, autotools obviously got it wrong.) Semantic Versioning is good, but how can this be verified and enforced?

autotools is a bear, but modern make systems are nicer, like cmake. There are more like Apache Maven, and you can use these systems to build and package distributables. I also like buildr, which is a Ruby alternative to Maven (compatible with maven).

How should library dependencies be handled? Can we download and install automatically? Will the versions and licenses be compatible?

Using cmake or maven can help you take care of dependencies, and maven can pull in remote dependencies or from a local server. cmake can do the same, but you do have to tell both these build systems what your dependencies are. Versioning and licensing is mostly your job, but if you were on a system like a Linux with a package manager, this could be mitigated.

Is the license for a given library compatible with your software? How can you tell?

You need due diligence here, because licensing is legal, not a software issue. You can see what the licenses are for various things in your system, and there are sites like FSF's License List and Comments which give FSF's interpretation of various licenses' compatibility with the GPL. Also, consult a lawyer.

What should you do if the library is abandoned before starting your software? What about after starting your software?

If it's abandoned before starting your software - evaluate to see if it's still useful to you, and check it's bug list. You could contact the maintainer and offer to take on the project.

If it's abandoned after - again, you can decide if it's worth it to continue, hopefully it's not so deeply embedded into your code that you couldn't switch it out with some effort. Of course, there are softwares where that is a problem and they get pegged to some old version. You can, though, also take the project under your wing and tailor it to your needs. Contact the maintainer.

Answer 3

1. Depends on the language. JRuby is performance-competitive with Ruby, but you would not want to run C on the JVM.

2. I'm not an expert in C, but if you google 'linked-list library C' you will find several results.

3. I trust libraries based on the backing (is it a library Google uses or an obscure one), how active is it (likely to be supported), and is it open-source (can I fix bugs/extend it myself).

4. Learning library APIs does take a lot of time. Fortunately, you only need to learn what you need to solve a particular problem, and many of the paradigms are the same across libraries (data structure names, file IO, etc).

5. Fix the bug and submit the fix. Hope it is included in the next release, then everyone will get it eventually. In the meantime people googling the same problem will find your fix.

6. Not sure what you are asking on this one.

7. Many possible answers here, Apache Maven is interesting. I don't think there is one 'correct' way to go about it.

8. Google the license. If your software will be commercial or closed-source it may limit your options.

9. Before starting the software you probably can defer deciding which libraries to use, but try to choose ones with high levels of activity. After starting software, hopefully you wrapped a risky library in a layer such that you can replace the implementation, otherwise you will have to start supporting it yourself.

Honestly, none of these are the reason software libraries don't solve all our problems. The reason is that problems are always changing and evolving. From moving to dynamic websites, mobile devices, or solving obscure problems in a domain, if there was a library to do it then companies wouldn't need to hire more than one programmer.

Answer 4

First of all, there's a lot more code reuse than most people give credit for, just not for every single application. Have you tried making a website using a modern CMS? "Merely a composition of high-quality software modules" is a perfect description.

There is an inherent coupling between the general applicability of an interface and its complexity. Let's say you wanted to make the container to end all containers. The interface couldn't be as simple as an array's because some people need O(1) insertions into the middle, some need fast lookup, some need persistent storage, some need space efficiency, etc. and they need a way to specify and operate upon those needs.

When you try to do too much in a library, your interface ends up with a complexity comparable to the programming language itself, so you may as well make narrowly-applicable interfaces and use the programming language as the glue, which is precisely what we have done.

Answer 5

The reason that software libraries can't solve all our problems is because software libraries just solve many (not all) of the problems of software developers. They don't solve the problems of software users!

Suppose your organization decides the world needs a really good 3D visual interactive aerospace design application? You, as the software developer, may well start with an existing 3D graphics library. For this exacting application you're almost certain to find that there is no library that does everything you need, you'll have to extend what already exists. And certainly there is no 'design a Boeing 747 or Airbus 380' library! It will take years of work to develop this application.

Suppose you are a cabinet maker, a customer comes in with a hand drawn sketch of a cabinet she wants created. You have a shelve full of woodworking books and a shop full of good woodworking tools.

The tools and books help, but you still have to make the cabinet!

Answer 6

So why don't software libraries solve all our problems? Why can't we write software as merely a composition of high-quality software modules?

People are trying to make this easier every day but there biggest hurdle I can think of is : Every 3rd party software library is still disjoint with how you store data.

How could a 3rd party PDF library know I want to create a PDF from my books table, author table and the sum of the prices for that author?

ORMs help tackle this but it's short of the sentient components you're hoping for.

Also, try to come up with a more concise question. Discussion is difficult with so many questions.

Answer 7

A difficulty that I don't see described clearly in other answers is how to find and exploit a relevant library efficiently. Even if all available libraries were searchable at http://code.google.com/hosting/ (and of course they are not) it would sometimes take longer for a programmer to complete that search process than to improvise an acceptable solution using the libraries that are already at hand.

Mathematicians have a similar problem to find math theorems that have already been proved. No one has yet perfected "mathematical knowledge management" software for that purpose. Aspects of the problem may be beyond the possibility of automating. The conceptual extreme case is Jorge Luis Borges's "Library of Babel" in which everything, as well as every possible variation on everything, is in the collection. Then the only unambiguous way to describe an item from the collection is to recreate it.

Edit: And where I just mentioned the possibility of having to choose between many variations on software libraries, a highly relevant example is different trade-offs between space (data storage) and time (CPU usage).

Edit 2: OK, my Library of Babel observation isn't mathematically correct. We can give names to the items in that catalog that interest us. But what I understand of Chaitin-Kolmogorov complexity leads me to believe that any such naming system leaves plenty of (perhaps unwanted) information resources whose most concise name is just as verbose as their complete content.