I'm just wondering—Are the programming language features unit tested?

  • Basic features, i.e. built-in types, operators, arrays, generics, etc.
  • The question applies to unit testing for runtime behaviour, not a compile-time, and not for a syntax validity.
  • The question doesn't apply to helping libraries (classes for strings, file, network, etc.)
  • The question applies to runtimes as well (CLR, JVM, etc.)
  • The question is about non-esoteric and non-code-golfing-specific languages.

If so—

  • What framework or language feature is used to?
  • What's the strategy for different environments, operating systems, multithreading etc?
  • What are the strategies for non-deterministic behaviours, i.e. garbage collector?
  • Are there any other kind of tests (other than unit tests)?
  • Programming languages are specifications (written in some document). They are often tested by some sample implementations (e.g. compilers or interpreters). – Basile Starynkevitch Nov 16 '16 at 7:06

Languages are not usually tested, but implementations are often tested very thoroughly – as very basic infrastructure, they have to work. However, unit testing a language is difficult since we can't test a feature in isolation. There is no way to delimit the “unit”. The different features of a language have rich and complicated interactions, and a valid program required for a test case will necessarily touch on many things that are not the goal of that specific test.

Such tests are integration tests: a whole program is executed and is expected to produce a particular output. Often, the language infrastructure includes a custom test runner to manage this. E.g. the TAP test result format originated with early Perl interpreter test cases.

The main source of new test cases is often regression tests: A user found a problem, and we want to make sure it never has to be fixed again. Some languages are effectively defined by their reference implementation (e.g. Ruby, Perl, Python). The language is whatever that reference implementation does. This makes tests a bit pointless. However, a complete test suite can be used to create compatible alternative implementations.

Many languages are defined by a specification. A specification cannot be tested or verified directly. Usually, the language already exists at least in prototypes or reference implementations before it is formalised. An interesting counter-example is Perl6, where the (informal) specification was translated into a suite of test cases before any implementation existed. This made it possibly to track the progress of potential implementations very visibly. One could see straight away which language features had yet to be implemented. Java also has a canonical test suite, but that is more related to the Java trademark than the Java language specification.

For parsers, fuzz testing may be very valuable. This doesn't really test the language, but makes sure the implementation is well behaved and uses defensive programming. Here, input to the implementation is mutated randomly. Usually, these changes should result in illegal syntax, so what we expect is an error message and a clean exit. If an error within the implementation is triggered, that test input is saved for human review as it likely indicates a possible bug.

| improve this answer | |
  • 4
    Unit testing is actually pretty easy in a compiler, since it’s basically a pipeline of pure functions. I typically write some unit tests for edge cases in tokenisation, parsing, typechecking, name resolution, optimisation, and interpretation or code generation. But beyond basic properties (e.g. “this program should produce these tokens” or “this type should fail to match this signature”), integration tests tend to have more value. – Jon Purdy Nov 14 '16 at 3:52
  • 1
    @JonPurdy Yes, you can unit-test the code in your compiler: codegen or optimization passes are easy to test; tests for the parser are also possible. But you can't test a for-loop or a variable assignment or recursion in isolation, that is: you can't really unit-test a language feature. – amon Nov 14 '16 at 10:04

Not the answer you're looking for? Browse other questions tagged or ask your own question.