Why do build tools use a scripting language different than underlying programming language?

Question

I've recently been using some build tools for a Nodejs project at work when I realized that most languages' main build tool/system use a different language than the underlying programming language itself.

For example, make does not use C or C++ to write scripts and ant (nor Maven) doesn't use Java as their language for scripting.

Newer languages like Ruby do use the same language for build tools like rake, which makes sense to me. But why hasn't this always been the case? What't the advantage to having a build tool that uses a different language from the underlying language?

Answer 1

The choice of what programming language to use to get anything done must depend on the specific features of that goal and not on the other tasks related to that project.

A build tool does a very specific job, and no matter what language you're using for the main project, a build tool is a software by itself.

Trying to tie the main project and its build tool could be a very bad decision. What you should need with a build tool is mainly a fast process of simple rules, with a fast management of files and IO.

If languages like ruby do use themselves to implement their build tool, is related more on the features of that language than on the assumed need to keep everything written with the same language.

Answer 2

It's not impossible to write a build tool that uses a language such as C or Java as their scripting languages. However, build tools benefit from using more declarative scripting languages. Imagine the pain and boilerplate of writing a makefile (or build.xml, or whatever) in C.

Build tools benefit from the use of DSLs, a task for which C isn't a particularly good choice. C is too general for a build tool, and too concerned with error-prone & low-level details. For example, you do not want to be concerned with explicit memory-management in a build tool! So even if using C-like syntax, you'd probably be using garbage collection in your scripting tool, at which point why not simply use a dedicated DSL?

It makes sense that Ruby can be used for this, since it's a higher-level, more declarative language than C or Java. Also see: Gradle, sbt.

Answer 3

Let’s give you a real world example.

About 15 years ago I worked on porting a large system written in C from Unix to Windows, it was about 3 million lines of code. To give you some idea of scale, it took over 24hr to compile on some of our unix systems (RS6000), windows could compile the system in about 4hrs.

(We also had 2 million line of code in our own interpreted language, but decided not to use our language for the build systems as it was never designed for file processing. Also we needed a build system to compile the C code that implemented our language.)

At the time the build system was written in a mix of shell scripts and make files, these were not portable to windows – therefore we decided to write our own build system.

We could have used C, however we decided to use python, there was few reasons. (We also rewrote our source code control system in python at the same time, this is was very intergraded with the build system, so object files for checked in modules could be shared by developers.)

• Most of our code could be built with a few simple rules (only a few thousand lines of python for all platforms, Windows, VMS, and 6 versions of Unix) that were drived from the naming conventions of the files.

• At the time RegEx was not very standard between C systems on different platforms, Python had built in RegEx.

• A few modules needed custom build steps, Python allowed class files to be loaded dynamically. We allowed a custom class to be used to build a module (lib), based on having the python file with a magic name in the folder. This was the killer reason to use python.

• We considered Java, but it was not shipping on all the platforms at that point.

(The UI for our source code control system used a web browser as that was portable across all the platform. This was 6 months before we had a internet connection. We had to download the browser over X25!)

Answer 4

The short answer to this question is that this is what a build tool is. A tool that automates the use of other tools, with the aim of building an end product from some source files. If we're writing the build script in the same language as the product itself, we're into the realms of language-specific tools, which would not be considered build tools in the same way.

This raises the question -- why don't compilers provide the sort of build automation that we're used to from tools like make? To which the answer is simply because make exists. The Unix philosophy of "do one thing, and do it well" suggests that it is not the responsibility of the compiler to provide this*. That said, I have personally been through my share of make headaches, and would be interested to try a compiler that provides it's own quote-unquote "native" build system.

For an example of a compiler designed this way, see Jon Blow's (as yet unreleased) Jai, a language intended as a C++ replacement. Links to talks and demos of the compiler are collected here. This language compiles to byte-code that runs within the compiler, with compilation to binary being a standard library provided function, accessible from the byte-code. The intention is to allow for both build automation and meta-programming within the language's native syntax.

_{* A look at Microsoft's Visual Studio will show you an example of the opposite philosophy. :)}

Answer 5

The build tool is first and foremost a tool, just like 'gcc', 'ls', 'awk' or 'git'. You feed the tool an input (filename, parameters, etc) and it will do some stuff accordingly.

All of these tools allow you to pass in your input in ways that should be simple enough to write and understand. In the particular case of build tools, the input is a recipe file, a file that describes how your project goes from the source files to the finished product.

If your recipe is as simple as passing in the folder that contains the project and the compiler to use, then a declarative "language" is enough. Once the recipe becomes more and more complicated, with more logic, it becomes troublesome to handle it with declarative languages and more general purpose languages are used.

It should now be apparent that there is no connection between the language used to write your build recipes and the languages used to write your product code simply because they have different purposes and requirements. There is no connection even between the language the build tool is written in and the "language" the build recipes should be written in. Always use the best tool for the job!