7

I've been developing my scientific-computation software for a few years in C++ and it performs pretty well on CFD and DEM simulations. However, it has about 12k lines of code without the dependencies. Any time I peek into another project, I'm always surprised how much lines are written even in a 'simple' project on rather specific tasks.

Examples:

  • cryptopp, encryptor with 100k lines (?!) of C++ (Git)

  • libxml2, XML-parser with 300k lines of C (Git)

I admit, these are well-written and useful libraries, but an XML-parser can be written in a few thousand lines. Furthermore, I was only looking for a lightweight 50 lines encryptor and found this monster...

Nevertheless, these are just two examples. I would like to obtain a generic answer, some sort of overview in this topic. I am not a software developer but an inquiring engineer.

13
  • 8
    While I don't know specifically why those libraries are so big, my guess would be that they are trying to solve a generic problem rather than a specific one. Writing a parser to deal with a specific piece of xml which only uses a small subset of the xml spec can be done in relatively few lines of code. Writing one that encompasses the entire spec and can ostensibly handle any piece of xml, adds robust error handling, good messaging, generic usage via templating, has automated tests, etc. is going to require a lot more code than the one-off parser to handle a 10 line xml file.
    – Becuzz
    Aug 23, 2017 at 12:25
  • 13
    "an XML-parser can be written in a few thousand lines" – How many XML parsers which support every feature of libxml have you written that you can confidently make this claim? Aug 23, 2017 at 12:54
  • 9
    You're making apples-to-suspension-bridges comparisons. Cryptopp contains several dozen algorithms, many of which occupy less than a few hundred lines.
    – Blrfl
    Aug 23, 2017 at 13:02
  • 4
    It is easy to underestimate the complexity of some software if you aren't familiar with the problem domain. E.g. LibXML doesn't just parse XML (which in itself has a nontrivial specification). It also supports different parsing modes, including stream-based and event-based parsing, an XPath implementation, DTD validation, tries to be very portable, can fetch remote resources via HTTP and FTP, and implements the Document Object Model. I guess a production-grade XML parser in C without all those important features would still clock in at 10kloc–50kloc.
    – amon
    Aug 23, 2017 at 13:16
  • 8
    "an XML-parser can be written in a few thousand lines"... and that's, my friends, is how fragile code is born.
    – Machado
    Aug 23, 2017 at 16:23

1 Answer 1

13

Libraries that are written to cater to a large, diverse group of users must be designed differently than project-specific code. You must make them more robust, handle a wider array of edge cases, and provide sufficient features and flexibility to make them useful to everyone.

You have to decide whether or not the additional bulk of code is worth the cost. If it is a well-maintained library, has good documentation and is intuitive, you might not care about the code bulk. And yes, I hate to admit it, but popularity does matter, if only because you can find a culture of people who already know and understand the library.

The truth is, while I probably use many libraries without even thinking about it (especially the .NET framework, which has considerable code bulk but also a vast array of features), I almost never have the need to maintain those libraries at the code level; other people do that for me. Since I don't typically program embedded devices (which have significant memory and processor constraints), I don't really think all that much about the code bulk.

I've used Newtonsoft.JSON for years. It has a simple, intuitive API, is very robust, and just works. I have no idea how large it is, but I'm sure its size is non-trivial. Its size has never been an issue for me, so I've never considered it a factor.

That said, in a significant number of cases after evaluating all of the alternatives, I choose custom code over libraries, because I just need some little thing done; I get to control the code, it's performance characteristics, etc., and the simplicity of doing that outweighs the cost of marshalling a giant library. I strongly favor simple systems; all other things being equal, less code is always better. But I also don't like to reinvent things that have already been done very well by other people, and there's only so much code I can write as one person.

Some technologies are based on an ecosystem of libraries, so if you choose one of these technologies, you're buying into a library philosophy. It wouldn't surprise me at all if Aurelia uses two hundred libraries of various types. In the financial world, we call this leverage; you're literally standing on the shoulders of giants.

Conversely, we've all seen "house of cards" projects where reams of code that doesn't do much except provide structure is stood up like an elaborate crystal cathedral. It's interesting code to look at, but I'm not convinced that a lot of that code carries its weight, in the sense that its benefits exceed the costs. A well-written library will avoid that.

Some ecosystems such as Javascript libraries have a culture of economy; libraries written in such a space tend to be small because they are subject to logistical constraints (i.e. browser bandwidth).

Some libraries are "opinionated," in the sense that you must buy into the philosophy of the library writer to use the library effectively. Caliburn Micro is an example of such a library; so is Angular and Knockout. Other libraries are more generic in nature. Genericity tends to breed more code.

In an era of multi-terabyte storage devices, multi-gigabyte memory and 15Mbit+ internet, getting the features you want in a library that you can trust is probably more important than code size.

1
  • 7
    There is also the complexity of the specification... a complex specification requires complex implementations to support it (XML). Aug 23, 2017 at 18:20

Not the answer you're looking for? Browse other questions tagged or ask your own question.