580

The Mars Curiosity rover has landed successfully, and one of the promo videos "7 minutes of terror" brags about there being 500,000 lines of code. It's a complicated problem, no doubt. But that is a lot of code, surely there was a pretty big programming effort behind it. Does anyone know anything about this project? I can only imagine it's some kind of embedded C.

16
  • 98
    Why would one assume there is only one language involved in the project.
    – rlperez
    Commented Aug 6, 2012 at 4:11
  • 5
    Good point, sure, it's probably got a breadth of technology associated with it. I want to know more about all of that :) Commented Aug 6, 2012 at 4:12
  • 3
    Which part? The spacecraft? The rover? Instruments? The ground system? As other comments indicate, there are probably several languages used in the different components. It's not out of the question that assembler was used for some of the time critical components.
    – GreenMatt
    Commented Aug 6, 2012 at 4:27
  • 71
    To be honest, when I saw the 500kloc figure I caught myself thinking "Only?" It could have been realistic had it been Haskell, but having read a bit about previous projects and their low level languages, this seemed way too low. The 2.5mio loc C code cited below are more believable. Commented Aug 6, 2012 at 6:02
  • 20
    A more interesting question that "in what language?" is "with what process?". It's the process that make the difference, and NASA has been using a rigorous one for decades now. Commented Aug 6, 2012 at 17:07

2 Answers 2

541

It's running 2.5 million lines of C on a RAD750 processor manufactured by BAE. The JPL has a bit more information but I do suspect many of the details are not publicized. It does appear that the testing scripts were written in Python.

The underlying operating system is Wind River's VxWorks RTOS. The RTOS in question can be programmed in C, C++, Ada or Java. However, only C and C++ are standard to the OS, Ada and Java are supported by extensions. Wind River supplies a tremendous amount of detail as to the hows and whys of VxWorks.

The underlying chipset is almost absurdly robust. Its specs may not seem like much at first but it is allowed to have one and only one "bluescreen" every 15 years. Bear in mind, this is under bombardment from radiation that would kill a human many times over. In space, robustness wins out over speed. Of course, robustness like that comes at a cost. In this case, it's a cool $200,000 to $500,000.

An Erlang programmer talks about the features of the computers and codebase on Curiosity.

34
  • 48
    JPL C language coding standards, specifically for embedded environments instead of "ground software" as they call it. lars-lab.jpl.nasa.gov/JPL_Coding_Standard_C.pdf Commented Aug 6, 2012 at 6:21
  • 81
    @Dynamic: It's such an important mission that NASA wouldn't risk it. Humans writing assembly make more errors, that's a measured fact.
    – MSalters
    Commented Aug 6, 2012 at 8:16
  • 23
    Compiled C code is machine code, assembly language is machine code, I don't see the difference. There isn't a huge performance difference when you get down to it.
    – Ramhound
    Commented Aug 6, 2012 at 11:43
  • 25
    NASA are extermely careful with their code. Everything (EVERYTHING) is done in the spec first and is repeatedly reviewed, checked and refined. When it is put into the life code stream it is almost a cut and paste of the spec's reference. The test scripts are given at least as much attention as the code is and no 'flashy' or clever code tricks are allowed unless they are critically needed.
    – Stefan
    Commented Aug 6, 2012 at 13:02
  • 103
    @Amarghosh: yeah, and see how well your cell phone works when it goes through a high-radiation environment such as outer space :) Commented Aug 6, 2012 at 15:02
193

The code is based on that of MER (Spirit and Opportunity), which were based off of their first lander, MPF (Sojourner). It's 3.5 million lines of C (much of it autogenerated), running on an RA50 processor manufactured by BAE and the VxWorks operating system. Over a million lines were hand coded.

The code is implemented as 150 separate modules, each performing a different function. Highly coupled modules are organized into components that abstract the modules they contain, and "specify either a specific function, activity, or behavior." These components are further organized into layers, and there are "no more than 10 top-level components."

Source: Keynote talk by Benjamin Cichy at 2010 Workshop on Spacecraft Flight Software (FSW-10), slides, audio, and video (starts with mission overview, architecture discussion at slide 80).


Someone on Hacker News asked "Not sure what means that most of the C code is auto generated. From what?"

I'm not 100% sure, although there probably is a separate presentation in that year or a different year that describes their auto-generation process. I know that it was a popular topic in general at the FSW-11 conference.

Simulink is a possibility. It's a MATLAB component popular among mechanical engineers, and therefore most navigation & control engineers, and allows them to 'code' and simulate things without thinking they're coding.

Model-based programming is definitely a thing that the industry is slowly becoming aware of, but I don't know how well it's catching on at JPL or if they would have chosen to use it when the project started.

The third and most likely possibility is for the communication code. With all space systems, you need to send commands to the flight software from the ground software, and receive telemetry from the flight software and process it with the ground software. Each command/telemetry packet is a heterogeneous data structure, and is is necessary that both sides are working from the exact same packet definition, and format the packet so it is correctly formatted on the one side, and parsed on the other side. This involves getting a whole lot of things right, including data type, size, and endianness (although the latter is usually a global thing; you could have multiple processors onboard with different endianness).

But that's just the surface. You need lots of repetitive code on both sides to handle things like logging, command/telemetry validation, limit checking, and error handling. And then you can do more sophisticated things. Say you have a command to set a hardware register value, and that value is sent back in telemetry in a particular packet. You could generate ground software that monitors that telemetry point to ensure that when this register value is set, eventually the telemetry changes to reflect the change. And of course, some telemetry points are more important than others (e.g. the main bus current) and are designated to come down in multiple packets, which involves extra copying on the flight side and data de-duplication on the ground side.

With all that, it's much easier (in my opinion) to write one collection of static text files (in XML, CSV, or some DSL/what-have-you), run them through a Perl/Python script, and presto! Code!

I do not work at JPL, so I cannot provide any detail that is not in the video, with one exception. I've heard that the autogenerated C code is written by Python scripts, and the amount of autocoding in a project varies greatly depending on who the FSW lead is.

13
  • 10
    This might shed some light on Wind River, the contractor who makes VxWorks: windriver.com/news/press/pr.html?ID=10901 I've read that NASA has a team of people whose job is to find as many bugs as they can in the control system code written by another team. The bug-finding team is rewarded for bugs they find and they are really quite good in finding arcane bugs. When a bug is found, a 5Y-type analysis is done to find out how the software dev process could be improved to eliminate the possibility of similar bugs in the future. A very painstaking and expensive process.
    – Jim Raden
    Commented Aug 6, 2012 at 17:47
  • 17
    @JimRaden When the direct cost of failure for a probe runs from several hundred million to several billion dollars and several years (if at all) for a redo attempt extreme paranoia in QA is justified. The indirect costs in the form of dozens/hundreds of grad students losing years of work and having to restart on their phd work and various new professors who were counting on data from it to supply their tenure track research is another major hit but much harder to quantify than the line items in the NASA budget. Commented Aug 6, 2012 at 18:41
  • 1
    What was the C auto-generated from? Please tell me that it was not Simulink. :-) Commented Aug 6, 2012 at 20:54
  • 2
    @William Payne The keynote states that some of it are autogenerated protocol encoding/decoding routines (for communication with earth), generated by python programs from XML descriptions.
    – nos
    Commented Aug 7, 2012 at 9:36
  • 1
    Automagically generating code from ICDs is kinda cool. I like the idea! I would have used YAML rather than XML, though. :-) Commented Aug 7, 2012 at 13:41

Not the answer you're looking for? Browse other questions tagged or ask your own question.