There are many comparative studies and available online when it comes to the runtime performance of applications built using one language or another. Some driven by corporations, some academic, some just personal experiment reports.

We also get a decent share of comparative studies on side-effects of a programming language and its tooling, like:

  • build times,
  • likelihood of post-production bug detection,
  • expressive power,
  • etc...

However, I've recently been more and more bummed out by the memory consumption of my programs more than anything else. This might come from the fact that while Moore's Law is on our side for raw performance, we have come to realize that other bottlenecks matter more. That, and I don't tend to update my hardware every so often, and I have some "old" (read 2005-2006 3.6GHz Pentium 4 with 4GB of RAM) that nowadays are hard-pressed to be useable for large applications without requiring me to go through great trouble to squeeze every bit of juice out of them (choice of OS, UI, tweaking of services and daemons, choice of applications to use for a task or another...). Quite honestly, sometimes I fire up top or procexp and weep at the sight of the memory used by the most innocent programs.

I can address this by keeping to push in the direction listed above, and essentially trying to limit myself and the programs I use (I have a dear love for cli programs for that reason, I guess), but I also cannot help but to think that maybe we're doing it wrong.

Modern Tools for Modern Needs

Of course, higher-level languages are arguably better and justify their worth of dead weight. Some design choices were made for good (or supposedly well-intended) reasons at the time, in many toolchains. Shared libraries, memory models, pre-processors, type-systems, etc... But some might be more viable than others with our modern hardware, and I'd be curious to read a few serious studies on the matter.

So, my question is, is there a pendant to the Benchmarks Game and others that focus on a comparison of the languages' base runtime memory consumption?

And even further, are there some studies that cross-reference this with other parameters (similar to what this article did, for instance, for other criteria, also based on the Benchmarks Game)?

closed as off-topic by gnat, Kilian Foth, Bart van Ingen Schenau, jwenting, user40980 Apr 30 '14 at 13:47

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking us to recommend a tool, library or favorite off-site resource are off-topic for Programmers as they tend to attract opinionated answers and spam. Instead, describe the problem and what has been done so far to solve it." – gnat, Kilian Foth, Bart van Ingen Schenau, jwenting, Community
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 3
    Why is the benchmarks game insufficient? It's probably the best resource there is, and it already covers memory consumption in detail. – Robert Harvey Jun 6 '13 at 16:44
  • @RobertHarvey: It does provide memory information, but it's not for the "base" runtime. Also, I find extracting information from the Benchmarks Game rather arcane (all the more credit for that article doing an amazing job with its data, though it's not the one I seek). – haylem Jun 6 '13 at 16:46
  • 1
    It might help the people trying to answer your question if you provided some information about what problem it is that you're trying to solve, with some specifics like your execution environment and your desired memory consumption. The answer is going to differ if you're writing software for an embedded environment (where the amount of memory used is important) versus a state-of-the-art desktop machine (where memory consumption is essentially inconsequential, unless the software system is extremely large). – Robert Harvey Jun 6 '13 at 16:46
  • 2
    How much memory consumption makes you weep? 30MBs for an inactive Chrome tab w/o extensions, 100MB for ATI's CCC, even 11MB for an inactive googletalk plugin, or 23MB for an inactive printer driver. These things, and many more. The chrome example is a bit out of the park as it's a more complex example, but the other ones already surprise me quite a bit. – haylem Jun 6 '13 at 17:00
  • 1
    let us continue this discussion in chat – Robert Harvey Jun 6 '13 at 17:04

I have found some partial information, so I will start to compile my findings in my own answer. Please don't let that stop you from contributing your own answers (or editing this one).

Existing Literature:

  • An Empirical Comparison of 7 Programming Languages - Prechelt (2000) [PDF]

    A bit dated, but covers some of the material I'm interested in and does give a view of the runtime memory use and expressiveness. Results may greatly differ now, but it's an interesting start.

  • Speed, Size and Dependendability of Programming Languages - Marceau (2009) [blog]

  • Code Used, Time Used Shapes from the Benchmarks Game [u32, u32q, u64, u64q]

    Though it doesn't cover runtime memory consumption, Marceau's work is more or less the kind of reference or empirical study I'd be willing to find for that critera, in terms of content and quality. A good example of what I'm after, just for different metrics. The second article is a follow-up found on the Benchmarks Game site and that was published shortly after (and which references) Marceau's work, with more recent screens and more languages, though still without runtime memory details. Each graph on these pages then leads to language-to-language comparison which do provide high-level memory info though.

  • Marceau's work is an exercise in story telling, and some of the stories don't make sense -- "Does introducing functional features kill performance?" ignores the simple fact that some of those "functional language" programs might not use functional features. The data was taken from a previous incarnation of the benchmarks game; and was initially used without understanding, so there were several cycles of correction after publication (check the comments). – igouy Jun 11 '13 at 17:44
  • For your "base runtime memory consumption" a simple comparison of "hello world" programs might be as good as you need. – igouy Jun 11 '13 at 17:59
  • @igouy: yes. Not doubt on that, but I was hoping not to have to experiment and document/maintain that myself :) In fact, even less than a hello world would be OK, as in some you wouldn't even be required to link to (or load) printing routines, for instance. (disabling compiler optimizations and others things might be advisable as well) – haylem Jun 11 '13 at 22:37
  • @igouy: regarding Marceau's work, I know, I've read the page, the comments, the updated Benchmark Game pages, and been in touch with him. The article is still a good reference in my opinion. The fact that it's imperfect doesn't take away its value and it still is in the direction of what I'd like to find (or recreate myself). – haylem Jun 11 '13 at 22:39
  • "but I was hoping not to have to experiment and document/maintain that myself" -- look at the measurements in the InternetArchive. Unfortunately for you I decided the memory measurements for Hello World were utterly misleading and stopped displaying them after 2005. – igouy Jun 12 '13 at 0:04

This is not answering the question per say, but perhaps changes the perspective a bit. I am bearing in mind the transcript from chat, to set the tone of this answer-comment that is sure to be subject of many down votes.

There are people, hardware providers, tool providers, and programmers who are concerned about efficiency. For the time being, it will be a growing concern for them and all of us. Those concerns are rooted in mobile devices, particularly the high powered, battery gobbling monsters with the biggest screens and the strongest radios.

To take one more step back, part of the reason we find ourselves in the situation we are in today, with comparatively massive frameworks, and some slight disregard for general efficiency, beyond the hardware improvements is legacy. Compatibility with legacy systems entrenches us with compatibility on top of compatibility. It is not so much the fault of a top level run-times, as they essentially are the same runtime acting quite efficient and performant when used in a different operating environment (e.g. Xbox, windows mobile pre 7/8/surface, java micro framework, etc).

Compare the extent of the compatibility a desktop possesses with its legacy software to the extent of compatibility a mobile device possesses.

With the mobile devices, the device producers make some effort to ensure some compatibility, but they have not made compatibility a core fundament. When the choice is between continuing to provide compatibility and moving the design of the mobile system forward, the mobile system moves forward.

For desktops, the opposite seems true. If a significant breaking change strikes the marketers or early adopters wrongly, it pushes needed features and needed redesign to the back room many times over. At some point I remember rumors to the effect that we as Windows users would find a completely and dramatically new file system with Windows XP, then in Vista, later the same for Seven, and finally again in Eight, but no, just incrementally improved since we first saw it on Windows2000? The new file-system sat around for a long time, was scrapped, and however rumors decide the story after that, I cannot say. That is probably the biggest known case, but I am positive it is not the only big case.

Even with the most recent tablets and mobile OS's, Microsoft who once shaped the market, is now intertwined in a death match with not only consumers, but a shadow of itself from the desktop department. The tablet had to have significantly interoperability with the desktop counterpart. No, it could not play perfectly with it, because of architecture differences, but also because of the archaic nature of the desktop underpinnings did it make significant sacrifices.

Now certainly, Windows is the easy target for any kind of criticism for this situation, but other platforms are far from "sin-free". There are a lot of relics lurking in the Linux ecosystem that I am sure cause great consternation for systematic improvement.

Economics play a big part into this equation; how we finance our computing and applications on one, and how they are financed on the other follow astoundingly different patterns. Where Wintel once strongly influenced obsolescence, Apple and Google have turned it into a near strict schedule. This is further off course, than I intended, so I will leave as it sits and let the readers take it from there.

If and when big providers change their obsolescence and pricing models, then they can start to move forward with larger scale changes at a more even rate. Those top-level frameworks that are driven by the highest order languages will shrink in a way, as they will be able to achieve their high-level task with a more low-level like efficiency because the inefficient intermediate compatibility and low-level layers will be drastically reduced, if not eliminated.

  • Doesn't really answer indeed, it's more like free-form thoughts adding to the bakground of the question :) Thanks and +1 for the insight, though. (Also, I want to clarify that I never intended to single out Microsoft systems as part of the culprits. Any OS as the same issue, if the system's memory model and the executable format allow it). – haylem Jun 7 '13 at 9:17
  • It certainly isn't my intent to poke at Microsoft, but they are the easiest case for most to see in this regard. Other big name, traditional providers are in the same boat, if even in perhaps a slightly different regard (industrial grade databases and networking equipment for instance; how many compromises do they carry forward that otherwise impedes significant improvement to their underlying product innovation and value). Even closer to home on the products each one of us supports, we carry that proverbial cross to one degree or another. – JustinC Jun 7 '13 at 9:24

Not the answer you're looking for? Browse other questions tagged or ask your own question.