Why those famous open source tools don't make them also a good-designed library? [closed]

Question

There are a lot of famous open source tools, for example: gcc, git, mercurial and etc.

However, they don't provide a nice API for developer.

Someone will say, they are command-line tools instead of library. But in my point of view, there is no conflict between a good interactive tools and a good library.

It can definitely firstly develop a well-designed library, and make a wrapper between cli-command to these API.

In addition, it can be make the logic in different layer of software become more clear which can bring a lot of convenience to future development.

The most beneficial point of this kind of architecture is it can prevent open source community reinvent wheels. For example, if git designed like that, there would be no libgit2. Then the works of bugfix and new feature implementation will become more easy and efficient.

Why the reality isn't like that? Is there any drawback of this kind of architecture?

Update 2019-04-12

In order to make my confusion more clear, I provide some example.

If git can implemented as below, it would be more awesome without much extra cost:

I don't think it will take more effort but it will implement the power both git and libgit2 provide at same time.

Answer 1

There are two levels of answering this: challenging your assumption in general, and addressing each project specifically. I will start with the general part.

The Unix Way

You are assuming that, because those projects have a rich commandline interface but no dynamic library, they don't have an API.

However, the Unix way of building applications is small tools that do exactly one thing well combined using shell scripts and shell features such as pipes, redirection, and substitution. Note that all three of the projects you mention were developed by Unix users for Unix users on a Unix platform for a Unix platform. (I'm using "Unix" in the general sense of "any Operating Environment inspired by the Unix Design Principles", i.e. I am including things like GNU or Cygwin.)

In other words, the rich commandline interface is the API you are asking about.

It can definitely firstly develop a well-designed library, and make a wrapper between cli-command to these API.

That is actually not quite true. In order to know what the interface should be, you really need multiple independenhttps://gcc.gnu.org/onlinedocs/jit/t clients. It is very, very hard to design a good API "out of thin air". Good APIs are extracted from multiple clients, in other words, they are discovered, not invented.

If you look at how the Java standardization process or the C++ standardization process works, for example, you will see that it often takes five years or even ten to design a good API … and those are people who do nothing else but design APIs 24/7, and it turns out they still make mistakes. Git went from idea to first production release in 12 days, if I remember correctly.

GCC

GCC is developed by the GNU Project, which is closely associated with the Free Software Foundation. GNU Projects in general, but GCC in particular, are as much a socio-political statement as they are software artifacts. GCC in particular was on a lot of platforms it supports actually much better than the platform vendor's own proprietary compiler.

The usage of the GPL License, the strict stance against supplying a C library / API, and the strict stance against a stable data format for intermediate representations, were all attempts on making sure that GCC stays Free Software and that everybody can enjoy the fruits of making GCC an even better compiler.

A C API or a file format for intermediate representations would have allowed someone to write a proprietary plugin or processor which makes use of GCC's hard-earned excellent portability, language support, backend support, and optimizations, while not giving their own optimization algorithms back to the community. Note that this fear was not unfounded since that is what actually happened. There were companies that forked GCC and added their own optimizations to it without disclosing their algorithms. The fact that in order to do that, they had to modify GCC and were thus violating the GPL was what made it possible to convince them to open up their algorithms (or in the worst case, stop them from distributing their product).

Now, you may have a different philosophy and may see nothing wrong with that, but that's not the point here: you are asking why the GCC developers made the choice to not offer an API, so it matters how they see it, and they see that as morally wrong and want to discourage it.

There's also a rather pragmatic reason: when GCC was designed in the 1980s, neither powerful IDEs with semantic code comprehension features nor JIT compilers were particularly mainstream, so two of the major drivers for wanting to embed a compiler into an application simply didn't exist.

Note that nowadays, where most platform vendors cooperate with and contribute to GCC (e.g. Intel, AMD, IBM, Apple, ARM, Oracle, Motorola, …) and there is an equally good competitor which is specifically designed to be used as a library inside IDEs etc. (Clang), there actually is libgccjit now, which solves one of the two major use cases. And GCC is extensible thru GCC plugins so provides some (under-documented, not very stable) API (with efforts, you can add your own optimization pass, but the GCC community and the FSF wants your GCC plugin to be free software, read the GCC runtime library exception).

Git

Git was written over a period of ~2 weeks by Linus Torvalds. It was written specifically for the Linux kernel development community. Many kernel developers had written sophisticated scripts (in shell, Perl, Python, whatever) to manage their source code, some based on GNU Diff and GNU Patch, some on CVS, some on Subversion, some on Bitkeeper. Linus himself, for example, was using a script called am (apply mailbox), which can parse the standard Unix mbox format for mailboxes, automatically find patch series that someone had posted to the linux-kernel mailinglist and automatically apply them in the correct order. So, it was in his own self-interest as well as the interest of his fellow developers, to make sure that those shell / Perl / Python scripts could be easily ported to Git.

Also, he intended Git to be very stupid, it should only act as a kind of versioned, content-addressable file system, and not more. (He called this the "plumbing". Note the connection to "pipes" in this analogy!) He envisioned that tools and GUIs (which he called the "porcelain") would be developed outside of Git, on top of the commandline interface. And that is exactly what happened. There was (and still is) Gitk written in Tcl using the Tk GUI toolkit, there was cogito, written in Perl, there still is git-stash which was originally a shell script. And git-am is a direct port of Linus's am script.

I mentioned above that designing APIs in a vacuum is hard, and that you need multiple independent clients in order to extract a good API. In Git's case, many of the top Linux kernel developers had written their own patch management systems specific to their own workflow. Linus Torvalds mostly integrates patches sent to him by other people, so had written am and other tools to help with that. Andrew Morton's kernel tree acted as a sort of filter and proving ground for new features, so he had developed tools to manage multiple rapidly evolving patch queues / stacks. Other developers used private CVS or Subversion repositories and had written their own tools on top of those. For a couple of years before the creation of Git, Linus and several of his top lieutenants used BitKeeper and had developed tools on top of that.

So, Linus could look at all these existing tools and how they interacted with the underlying version control system (in case of the ones built on top of CVS, Subversion, and BitKeeper) or what they would require from a VCS in order to keep working (in case of the ones that were only using the file system), and extract an API from that. And it turned out that all of those tools were scripts, and they needed an API that was based on environment variables, pipes, redirections, sockets, files, command line arguments, and command substitution.

Note that today, Git has changed: Git is no longer just the "plumbing", it also contains lots of "porcelain" as well. Also, note that a lot of its original shell, Perl, Python, Tcl, and other scripts, have been re-written in C. Not, because the developers necessarily deem that to be a better design, but in significant part because it makes it easier to port Git to Operating Environments that don't come with a shell interpreter, Perl interpreter, Tcl interpreter, Python interpreter, and a full set of Unix tools (diff, patch, awk, sed, tr, …) pre-installed as well as Operating Systems where processes, pipes, and redirections are slow and expensive – in other words, Windows.

Also, note that today there is libgit2.

Mercurial

Note that Mercurial actually does provide an API for Extensions. This is a Python API, not a C API, but it is an API.

While Mercurial does not guarantee stability of this API, all API changes are meticulously documented.

Answer 2

Few open source utilities were originally developed to be good libraries. They were instead developed to be good utilities, and any library functionality grew as the utility needed it to.

Everything takes effort to achieve. Providing multiple layers of interfaces while arguably more flexible also imposes a burden a maintenance and curation. Many open source efforts are not developed by large focused teams. They are generally developed by small teams, or by fragmented teams.

• small teams lack the man power to support too many interfaces
• fragmented teams lack the vision to develop and maintain, cohesive and clean interfaces

There are of course exceptions. Some open source projects have a large community of individuals willing to work toward a road map of features that they will never themselves need.

However that is work, and not many developers are paid enough to do work. They might work to make life easier on themselves or they don't work and might instead have fun - only as a consequence of that fun producing code.

All of that being said, the best way to answer your question is for you to go out and join/start an open source effort. Before long you will have answered your own question.

Answer 3

Hooking up to a command line program is straight forward in just about any useful programming language.

Hooking up to a library means you have to either use the same language, often the same compiler, or someone has to expend the effort to create cross-language bindings.

If you want to make a standalone utility integratable into larger works, exposing the entire functionalality thru the command line is a good way to make that achievable.