Why are standard libraries not programming language primitives? [closed]

Question

I was thinking why are there (in all programming languages I have learned, such as C++, Java, Python) standard libraries like stdlib, instead of having similar "functions" being a primitive of the language itself.

Answer 1

Allow me to expand somewhat on @Vincent's (+1) good answer:

Why couldn't the compiler simply translate a function call into a set of instructions?

It can, and does so via at least two mechanisms:

• inlining a function call — during translation, the compiler can replace a source code call with its implementation directly inline instead of making an actual call to the function.  Still the function needs to have an implementation defined somewhere and that can be in the standard library.

• intrinsic function — intrinsics are functions that the compiler has been informed of without necessarily finding the function in a library.  These are usually reserved for hardware features that are not practically accessible in any other way, being so simple that even the overhead of a call to assembly language library function is considered high.  (The compiler can generally only automatically inline source code in its language, but not assembly functions, which is where intrinsic mechanism comes in.)

Still these being said, the best option sometimes is for the compiler to translate a function call in the source language into a function call in the machine code.  Recursion, virtual methods, and sheer size are some reasons that inlining is not always possible/practical.  (Another reason is intent of the build, such as separate compilation (object modules), separate load units (e.g. DLLs)).

There's no real advantage to making most standard library functions intrisics either (that would hard code a lot more knowledge into the compiler for no real advantage), so a machine code call again is often most appropriate.

C is a notable language that arguably omitted other explicit language statements in favor standard library functions.  Though libraries pre-existed, this language made a shift to doing more work from standard library functions and less as explicit statements in the grammar of the language.  IO in other languages, for example, was frequently given its own syntax in the form of various statements, whereas the C grammar does not define any IO statements, simply instead deferring to its standard library to provide that, all accessible via function calls, which the compiler already knows how to do.

Answer 2

This is simply to keep the language itself as simple as possible. You need to distinguish between a feature of the language, such as a type of loop or ways to pass parameters to functions and so on, and common functionality that most applications need.

Libraries are functions that may be useful to many programmers so they are created as reusable code that can be shared. The standard libraries are designed to be very common functions that programmers typically need. This way the programming language is immediately useful to a wider range of programmers. The libraries can be updated and extended without changing the core features of the language itself.

Answer 3

In addition to what the other answers have already said, putting standard functions into a library is separation of concerns:

• It's the compiler's job to parse the language and generate code for it. It's not the compiler's job to contain anything that can already be written in that language and provided as a library.

• It's the standard library's (the one that's always implicitly available) job to provide core functionality that's needed by virtually all programs. It's not the standard library's job to contain all the functions that might be useful.

• It's the job of optional standard libraries to provide auxiliary functionality that many programs can do without, but which are still quite basic and also essential for many applications to warrant shipping with standard environments. It's not the job of those optional libraries to contain all reusable code that's ever been written.

• It's the job of user libraries to provide collections of useful reusable functions. It's not the job of user libraries to contain all the code that's ever been written.

• It's the job of an application's source code to provide the remaining bits of code that are really only relevant to that one application.

If you want a one-size-fits-all software, you get something insanely complex. You need to modularize to get the complexity down to manageable levels. And you need to modularize to allow partial implementations:

• The threading library is worthless on the single-core embedded controller. Allowing the language implementation for this embedded controller to just not include the pthread library is just the right thing to do.

• The math library is worthless on the micro-controller that doesn't even have an FPU. Again, not being forced to provide functions like sin() makes life a whole lot easier for the implementators of your language for that micro-controller.

• Even the core standard library is worthless when you are programming a kernel. You cannot implement write() without a syscall into the kernel, and you cannot implement printf() without write(). As a kernel programmer, it's your job to provide the write() syscall, you cannot just expect it to be there.

A language that does not allow for such omissions from the standard libraries is simply not suited for many tasks. If you want your language to be flexibly usable in uncommon environments, it must be flexible in what standard libraries are included. The more your language relies on standard libraries, the more assumptions it makes on its execution environment, and thus restricts its use to environments that provide these prerequisites.

Of course, high level languages like python and java can make a lot of assumptions on their environment. And they tend to include many, many things into their standard libraries. Lower level languages like C provide much less in their standard libraries, and keep the core standard library much smaller. That's why you find a working C compiler for virtually any architecture, but may not be able to run any python scripts on it.

Answer 4

One big reason compilers and standard libraries are separate are because they serve two different purposes (even if they're both defined by the same language spec): the compiler translates higher-level code into machine instructions, and the standard library provides pre-tested implementations of commonly-needed functionality. Compiler writers value modularity just like other software developers do. In fact, some of the early C compilers further split the compiler into separate programs for pre-processing, compiling, and linking.

This modularity gives you a bunch of advantages:

• It minimizes the amount of work needed when supporting a new hardware platform, since most of the standard library code is hardware-agnostic can be re-used.
• A standard library implementation can be optimized in different ways (for speed, for space, for resource usage, etc). Many early computing systems only had one compiler available, and having a separate standard library meant developers could swap implementations to suit their needs.
• The standard library functionality doesn't even have to exist. When writing bare-metal C code for instance, you have a full-featured compiler but most of the standard library functionality isn't there and some things like file I/O aren't even possible. If the compiler was required to implement this functionality, then you couldn't have a standards-conforming C compiler on some of the platforms where you need it the most.
• On early systems, compilers were frequently developed by the company that designed the hardware. Standard libraries were frequently provided by the OS vendor, since they often required access to functionality (like system calls) specific to that software platform. It was impractical for a compiler writer to have to support all of the different combinations of hardware and software (there used to be a whole lot more variety in both hardware architecture and software platform).
• In high-level languages, a standard library can be implemented as a dynamically-loaded library. One standard library implementation can then be used by multiple compilers and/or programming languages.

Historically speaking (at least from C's perspective), the original, pre-standardization versions of the language didn't have a standard library at all. OS vendors and third parties would often provide libraries full of commonly-used functionality, but different implementations included different things and they were largely incompatible with each other. When C was standardized, they defined a "standard library" in an attempt to harmonize these disparate implementations and improve portability. The C standard library developed separate from the language, like the Boost libraries have for C++, but were later integrated into the language spec.

Answer 5

Additional corner-case answer: Intellectual property management

Notable example is implementation of Math.Pow(double, double) in .NET Framework which was purchased by Microsoft from Intel and remains undisclosed even if the framework went open-source. (To be precise, in the above case it is an internal call rather than a library but the idea holds.) A library separated from the language itself (theoretically also a subset of standard libraries) can give the language backers more flexibility in drawing the line between what to keep transparent and what has to remain undisclosed (due to their contracts with 3rd parties or other IP-related reasons).

Answer 6

As a language designer myself, I'd like to echo some of the other answers here, but provide it through the eyes of someone who is building a language.

An API is not finished when you are done adding everything you can into it. An API is finished when you're done taking everything you can out of it.

A programming language has to be specified using some language. You have to be able to convey the meaning behind any program written in your language. This language is very hard to write, and even harder to write well. In general, it tends to be a very precise and well structured form of English used to convey meaning not to the computer, but to other developers, especially those developers writing compilers or interpreters for your language. Here's an example from the C++11 spec, [intro.multithread/14]:

The visible sequence of side effects on an atomic object M, with respect to a value computation B of M, is a maximal contiguous sub-sequence of side effects in the modification order of M, where the first side effect is visible with respect to B, and for every side effect, it is not the case that B happens before it. The value of an atomic object M, as determined by evaluation B, shall be the value stored by some operation in the visible sequence of M with respect to B. [ Note: It can be shown that the visible sequence of side effects of a value computation is unique given the coherence requirements below. —end note ]

Blek! Anyone who has taken the plunge into understanding how C++11 handles multithreading can appreciate why the wording has to be so dang opaque, but that doesn't forgive the fact that it is... well... so opaque!

Contrast that with the definition of std::shared_ptr<T>::reset, in the library section of the standard:

template <class Y> void reset(Y* p);

Effects: Equivalent to shared_ptr(p).swap(*this)

So what's the difference? In the language definition part, the writers cannot assume that the reader understands the language primitives. Everything must be specified carefully in English prose. Once we get to the library definition part, we can use the language to specify the behavior. This is often far easier!

In principle, one could have a smooth build up from primitives at the start of the spec document, all the way up through defining what we would think of as "standard library features", without having to draw a line between "language primitives" and "standard library" features. In practice, that line proves enormously valuable to draw because it lets you write some of the most complex parts of the language (such as those which must implement algorithms) using a language designed to express them.

And we do indeed see some blurry lines:

• In Java, java.lang.ref.Reference<T> may only be subclassed by the standard library classes java.lang.ref.WeakReference<T> java.lang.ref.SoftReference<T> and java.lang.ref.PhantomReference<T> because the behaviors of Reference are so deeply entwined with the Java language specification that they needed to put some restrictions into the portion of that process implemented as "standard library" classes.
• In C#, there is a class, System.Delegate which encapsulates the concept of delegates. Despite its name, it is not a delegate. It is also an abstract class (cannot be instantiated) that you cannot create derived classes from. Only the system can do it through features written into the language specification.

Answer 7

Bugs and debugging.

Bugs: All software has bugs, your standard library has bugs and your compiler has bugs. As a user of the language it is much easier to find and workaround such bugs when they're in the standard library as opposed to in the compiler.

Debugging: It's much easier for me to see a stack trace of a standard library and give me some sense of what might be going wrong. Because that stack trace has code I understand. Ofcourse you can do dig deeper and you can also trace your intrinsic functions, but it's a lot easier if it's in a language you use all the time from day to day.

Answer 8

This is an excellent question!

State of the Art

The C++ Standard, for example, never specifies what should be implemented in the compiler or in the standard library: it just refers to the implementation. For example, reserved symbols are defined both by the compiler (as intrinsics) and by the standard library, interchangeably.

Yet, all C++ implementations that I know of will have the minimum possible number of intrinsics provided by the compiler, and as much as possible provided by the standard library.

Thus, while it is technically feasible to define the standard library as intrinsic functionality in the compiler, it seems rarely used in practice.

Why?

Let's consider the idea of moving some piece of functionality from the standard library to the compiler.

Advantages:

• Better diagnostics: intrinsics can be special-cased.
• Better performance: intrinsics can be special-cased.

Disadvantages:

• Increased compiler mass: each special-case adds complexity to the compiler; complexity increases maintenance costs, and the likelihood of bugs.
• Slower iteration: changing the implementation of the functionality requires changing the compiler itself, making it harder to create just a small library (outside of std) to experiment.
• Higher bar to entry: the more expensive/more difficult it is to change something, the less people are likely to jump in.

This means that moving something to the compiler is expensive, now and in the future, and therefore it requires a solid case. For some pieces of functionality, it is necessary (they cannot be written as regular code), however even then it pays to extract minimal and generic pieces to move to the compiler and build atop them in the standard library.

Answer 9

This is meant as an addition to the existing answers (and is too long for a comment).

There are at least two other reasons for a standard library:

Barrier to Entry

If a particular language feature is in a library function and I want to know how it works, I can just read the source for that function. If I want to submit a bug report/patch/pull request, it's not generally too difficult to code a fix and test case(s). If it's in the compiler, I have to be able to dig into the internals. Even if it's in the same language (and it should be, any self-respecting compiler should be self-hosted) compiler code is nothing like application code. It may take forever to even find the correct files.

You're cutting yourself off from a lot of potential contributors if you go that route.

Hot code loading

Many languages offer this feature to one degree or another, but it would be enormously complicated to hot reload the code that's doing the hot reloading. If the SL is separate from the runtime it can be reloaded.

Answer 10

This is an interesting question but there are many good answers already given, so I won't attempt a complete one.

However, two things that I don't think have gotten enough attention:

First is that the whole thing is not super clear cut. It's a bit of a spectrum exactly because there are reasons to do things differently. As an example, compilers often know about standard libraries and their functions. Example of the example: C's "Hello World" function - printf - is the best one I can think of. It's a library function, it sort-of has to be, as it's very platform dependent. But it's behaviour (implementation defined) needs to be known by the compiler in order to warn the programmer about bad invocations. This isn't particularly neat, but was seen as a good compromise. Incidentally, this is the real answer to most of the "why this design" questions: a lot of compromise and "seemed like a good idea at the time". Not always the "this was the clear way to do it" or "look: an example from on of the far ends of the spectrum" as often given by the proponents of that choice. (Not that those answers are not relevant).

Second is that it allows the standard library not to be all that standard. There a lot of situations that a language is desirable but the standard libraries that usually accompany them are not both practical and desirable. This is most commonly the case with systems programming languages like C, on non-standard platforms. For example, if you have a system without an OS or a scheduler: you aren't going to have threading.

With a standard library model (and threading being supported in it) this can be handled cleanly: the compiler is pretty much the same, you can reuse the bits of the libraries that apply, and anything that doesn't you can remove. If this is baked into the compiler things start to get messy.

For example:

• You can't be a compliant compiler.

• How would you indicate your deviation from the standard. Note there is usually some form of import/include syntax you can have fail i.e. pythons's import or C's include that easily points to the problem if there is anything missing in the standard library model.

Also similar problems apply if you want to tweak or extend 'library' functionality. This is far more common than you might think. Just to stick with threading: windows, linux and some-exotic-network-processing-units all do threading quite differently. While the linux/windows bits might be fairly static and be able to use an identical API, the NPU stuff will change with the day of the week and the API with it. Compilers would quickly deviate as people decided which bits they need to support/could-do-with-out quite rapidly if there was no way to split this sort of thing out.