-1

There are a lot of famous open source tools, for example: gcc, git, mercurial and etc.

However, they don't provide a nice API for developer.

Someone will say, they are command-line tools instead of library. But in my point of view, there is no conflict between a good interactive tools and a good library.

It can definitely firstly develop a well-designed library, and make a wrapper between cli-command to these API.

In addition, it can be make the logic in different layer of software become more clear which can bring a lot of convenience to future development.

The most beneficial point of this kind of architecture is it can prevent open source community reinvent wheels. For example, if git designed like that, there would be no libgit2. Then the works of bugfix and new feature implementation will become more easy and efficient.

Why the reality isn't like that? Is there any drawback of this kind of architecture?

Update 2019-04-12

In order to make my confusion more clear, I provide some example.

If git can implemented as below, it would be more awesome without much extra cost:

enter image description here

I don't think it will take more effort but it will implement the power both git and libgit2 provide at same time.

closed as primarily opinion-based by gnat, Pieter B, Bart van Ingen Schenau, Robert Harvey Apr 10 at 16:04

Many good questions generate some degree of opinion based on expert experience, but answers to this question will tend to be almost entirely based on opinions, rather than facts, references, or specific expertise. If this question can be reworded to fit the rules in the help center, please edit the question.

  • 1
    I am sorry to disagree, but git api is incredibly good :) – Gonzalo.- Apr 10 at 0:59
  • Why the reality isn't like that? -- Because there isn't always time and resources available to create such an API. – Robert Harvey Apr 10 at 2:08
  • What would you expect a gcc library to do? – Bryan Oakley Apr 10 at 2:45
  • 1
    In the case of GCC it used to also be a deliberate effort to block non-open-source plugins. – immibis Apr 10 at 4:04
  • 1
    Your question was quite clear, the clarification is a simple restatement, the answer is still the same. Your design has benefits, it also has a price. No one is preventing you from making that choice on your own projects. However open source projects often have a set of needs not served by this implementation or better served by another implementation, that is why they are not implemented this way. Should their needs change sufficient developers will eventual change the system to support it. Will you be one of those developers? Otherwise do not complain. – Kain0_0 Apr 12 at 7:10
13

There are two levels of answering this: challenging your assumption in general, and addressing each project specifically. I will start with the general part.

The Unix Way

You are assuming that, because those projects have a rich commandline interface but no dynamic library, they don't have an API.

However, the Unix way of building applications is small tools that do exactly one thing well combined using shell scripts and shell features such as pipes, redirection, and substitution. Note that all three of the projects you mention were developed by Unix users for Unix users on a Unix platform for a Unix platform. (I'm using "Unix" in the general sense of "any Operating Environment inspired by the Unix Design Principles", i.e. I am including things like GNU or Cygwin.)

In other words, the rich commandline interface is the API you are asking about.

It can definitely firstly develop a well-designed library, and make a wrapper between cli-command to these API.

That is actually not quite true. In order to know what the interface should be, you really need multiple independenhttps://gcc.gnu.org/onlinedocs/jit/t clients. It is very, very hard to design a good API "out of thin air". Good APIs are extracted from multiple clients, in other words, they are discovered, not invented.

If you look at how the Java standardization process or the C++ standardization process works, for example, you will see that it often takes five years or even ten to design a good API … and those are people who do nothing else but design APIs 24/7, and it turns out they still make mistakes. Git went from idea to first production release in 12 days, if I remember correctly.

GCC

GCC is developed by the GNU Project, which is closely associated with the Free Software Foundation. GNU Projects in general, but GCC in particular, are as much a socio-political statement as they are software artifacts. GCC in particular was on a lot of platforms it supports actually much better than the platform vendor's own proprietary compiler.

The usage of the GPL License, the strict stance against supplying a C library / API, and the strict stance against a stable data format for intermediate representations, were all attempts on making sure that GCC stays Free Software and that everybody can enjoy the fruits of making GCC an even better compiler.

A C API or a file format for intermediate representations would have allowed someone to write a proprietary plugin or processor which makes use of GCC's hard-earned excellent portability, language support, backend support, and optimizations, while not giving their own optimization algorithms back to the community. Note that this fear was not unfounded since that is what actually happened. There were companies that forked GCC and added their own optimizations to it without disclosing their algorithms. The fact that in order to do that, they had to modify GCC and were thus violating the GPL was what made it possible to convince them to open up their algorithms (or in the worst case, stop them from distributing their product).

Now, you may have a different philosophy and may see nothing wrong with that, but that's not the point here: you are asking why the GCC developers made the choice to not offer an API, so it matters how they see it, and they see that as morally wrong and want to discourage it.

There's also a rather pragmatic reason: when GCC was designed in the 1980s, neither powerful IDEs with semantic code comprehension features nor JIT compilers were particularly mainstream, so two of the major drivers for wanting to embed a compiler into an application simply didn't exist.

Note that nowadays, where most platform vendors cooperate with and contribute to GCC (e.g. Intel, AMD, IBM, Apple, ARM, Oracle, Motorola, …) and there is an equally good competitor which is specifically designed to be used as a library inside IDEs etc. (Clang), there actually is libgccjit now, which solves one of the two major use cases. And GCC is extensible thru GCC plugins so provides some (under-documented, not very stable) API (with efforts, you can add your own optimization pass, but the GCC community and the FSF wants your GCC plugin to be free software, read the GCC runtime library exception).

Git

Git was written over a period of ~2 weeks by Linus Torvalds. It was written specifically for the Linux kernel development community. Many kernel developers had written sophisticated scripts (in shell, Perl, Python, whatever) to manage their source code, some based on GNU Diff and GNU Patch, some on CVS, some on Subversion, some on Bitkeeper. Linus himself, for example, was using a script called am (apply mailbox), which can parse the standard Unix mbox format for mailboxes, automatically find patch series that someone had posted to the linux-kernel mailinglist and automatically apply them in the correct order. So, it was in his own self-interest as well as the interest of his fellow developers, to make sure that those shell / Perl / Python scripts could be easily ported to Git.

Also, he intended Git to be very stupid, it should only act as a kind of versioned, content-addressable file system, and not more. (He called this the "plumbing". Note the connection to "pipes" in this analogy!) He envisioned that tools and GUIs (which he called the "porcelain") would be developed outside of Git, on top of the commandline interface. And that is exactly what happened. There was (and still is) Gitk written in Tcl using the Tk GUI toolkit, there was cogito, written in Perl, there still is git-stash which was originally a shell script. And git-am is a direct port of Linus's am script.

I mentioned above that designing APIs in a vacuum is hard, and that you need multiple independent clients in order to extract a good API. In Git's case, many of the top Linux kernel developers had written their own patch management systems specific to their own workflow. Linus Torvalds mostly integrates patches sent to him by other people, so had written am and other tools to help with that. Andrew Morton's kernel tree acted as a sort of filter and proving ground for new features, so he had developed tools to manage multiple rapidly evolving patch queues / stacks. Other developers used private CVS or Subversion repositories and had written their own tools on top of those. For a couple of years before the creation of Git, Linus and several of his top lieutenants used BitKeeper and had developed tools on top of that.

So, Linus could look at all these existing tools and how they interacted with the underlying version control system (in case of the ones built on top of CVS, Subversion, and BitKeeper) or what they would require from a VCS in order to keep working (in case of the ones that were only using the file system), and extract an API from that. And it turned out that all of those tools were scripts, and they needed an API that was based on environment variables, pipes, redirections, sockets, files, command line arguments, and command substitution.

Note that today, Git has changed: Git is no longer just the "plumbing", it also contains lots of "porcelain" as well. Also, note that a lot of its original shell, Perl, Python, Tcl, and other scripts, have been re-written in C. Not, because the developers necessarily deem that to be a better design, but in significant part because it makes it easier to port Git to Operating Environments that don't come with a shell interpreter, Perl interpreter, Tcl interpreter, Python interpreter, and a full set of Unix tools (diff, patch, awk, sed, tr, …) pre-installed as well as Operating Systems where processes, pipes, and redirections are slow and expensive – in other words, Windows.

Also, note that today there is libgit2.

Mercurial

Note that Mercurial actually does provide an API for Extensions. This is a Python API, not a C API, but it is an API.

While Mercurial does not guarantee stability of this API, all API changes are meticulously documented.

  • Apple explicitly got rid of gcc and sponsored work on clang and llvm. Because LLVM can be used to compile JavaScript or OpenGL and OpenCL at runtime. – gnasher729 Apr 10 at 20:04
  • Yes, I know libgit2. I was a excellent library, However, it is independent from git. Why those software like julia use libgit2 instead of git? – davmos Apr 12 at 5:58
  • @davmos: Git offers a Unix-friendly shell API. JGit offers a Java-ecosystem-friendly Java API. libgit2 offers a C-friendly C API. You would choose betwee the three depending on what kind of API is easiest to access from the particular language you are using. Julia libraries can be easily written in Julia, and almost as easily be written in C (or languages that interoperate with C), so libgit2 is the simplest choice. – Jörg W Mittag Apr 12 at 6:04
  • For GCC, I am not quite understand. In my point of view, it can be resolved through legal method, without bring any chaos in the software which bring inconvenience to anyone. – davmos Apr 12 at 6:07
  • 1
    @davmos: "And clear and stable intermediate representation is good for newbie to learn and there will be more people who are able to involve in the development of GCC." – A stable intermediate representation also allows you to write a tool that is independent of GCC (thus not covered by the GPL) which reads that intermediate representation, performs optimizations on it, and feeds the optimized form of the intermediate representation back to GCC. In other words, it allows you to write a proprietary plugin, which is exactly what the developers of GCC wanted to prevent. – Jörg W Mittag Apr 12 at 6:42
6

Few open source utilities were originally developed to be good libraries. They were instead developed to be good utilities, and any library functionality grew as the utility needed it to.

Everything takes effort to achieve. Providing multiple layers of interfaces while arguably more flexible also imposes a burden a maintenance and curation. Many open source efforts are not developed by large focused teams. They are generally developed by small teams, or by fragmented teams.

  • small teams lack the man power to support too many interfaces
  • fragmented teams lack the vision to develop and maintain, cohesive and clean interfaces

There are of course exceptions. Some open source projects have a large community of individuals willing to work toward a road map of features that they will never themselves need.

However that is work, and not many developers are paid enough to do work. They might work to make life easier on themselves or they don't work and might instead have fun - only as a consequence of that fun producing code.

All of that being said, the best way to answer your question is for you to go out and join/start an open source effort. Before long you will have answered your own question.

  • I have update more post, please have a see. – davmos Apr 12 at 6:36
4

Hooking up to a command line program is straight forward in just about any useful programming language.

Hooking up to a library means you have to either use the same language, often the same compiler, or someone has to expend the effort to create cross-language bindings.

If you want to make a standalone utility integratable into larger works, exposing the entire functionalality thru the command line is a good way to make that achievable.

  • I have update my post. please have a look. – davmos Apr 12 at 6:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.