10

Has memory and processing power increased over time around basically in the same rate, or can we say that one or the other has increased more? Has cost decreased in the same rate? And how about the near future?

I am aware this question is not directly related to programming, but I thought it would be better to ask it here for some reasons:

  1. I work as a programmer, not a computer science theorist (so I probably would not find it very helpful the kind of question I would receive from the Theoretical Computer Science site, for instance).
  2. I prefer answers based upon a programmers real experience.
  3. And I am worried about real programming-related issues, specially the functional x imperative debate.

About the 3rd item: currently we've being seeing a lot of arguments about how the increase of using functional programming techniques is related to the now common multi-core machines, and how the free lunch is over [pdf].

But it seems to me that the functional paradigm is much more memory consuming (but this is an opinion, not a fact), and I have not seem many arguments about this future increase in memory needs.

2
  • 2
    It's a matter of computer manufacture, and how many transistors fit on a chip. Moore's law is a general observation (and, to some extent, a self-fulfilling prophecy), and has no theoretical basis. Jan 19, 2011 at 18:05
  • 2
    I think #3 and the last half of your question are very related to programming - can you rephrase the question to make that primary?
    – Nicole
    Jan 19, 2011 at 20:34

6 Answers 6

11

It's important to note that Moore's Law does not talk about processor speeds. It talks about transistor density.

In fact, we hit the wall on clock rates quite a long time ago, and clocks have generally been decreasing since the P4. There have been additional performance gains with stuff like OOO execution and other mechanisms to exploit ILP, but these days the extra density is going towards more cache and more cores, rather than straightforwardly boosting the speed of each core.

Memory, on the other hand, is much more straightforward to implement - a higher transistor density basically means you can pack more memory cells into the same package, without having to do any complex redesign.

Basically, memory scales exceptionally well with increasing transistor density. Processors, not so much.

1
  • Thanks, that's more in the spirit of what I was asking about.
    – rsenna
    Jan 20, 2011 at 12:45
6

Not strictly an answer, but ...

Your question implies that you are wondering whether functional programming will become unavoidable due to limitations in moore's law. Perhaps that's besides the point. The main driver behind new technologies in programming is not performance, or we would all be writing assembly. The main driver is complexity. New programming techniques and technologies become popular because they make it easier to build bigger systems.

In other words, if functional programming takes over, it won't be because it offers better performance, it will be because it lets you build bigger apps faster. And that has nothing to do with moore's law.

2
  • I am not that sure of the irrelevance of performance as you seem to be, and I think the real scenario is much more complex (take for instance web applications vs. games; they have very distinct performance requirements). And even if performance is almost irrelevant, it may be still a big technological drive over time - Darwin proved that for biological replicators, and I guess the same applies for technology. But, of course, I may be wrong.
    – rsenna
    Jan 20, 2011 at 12:15
  • Performance is not irrelevant to programming, but the majority of programming tasks are not highly performance-sensitive, and those gravitate to tools that optimize for programmability, not performance. Jan 20, 2011 at 19:48
3

So Moore's Law specifically states that the density of transistors will double about every 18 months. What this translates into effectively is faster processors and more RAM. Let's look at it from a few comparisons.

1996 A top of the line desktop had about a 300 MHz Processor, 32 MBs of RAM and a 2GB hard drive. Computers were just becoming powerful enough that the MPEG audio format (MP3) could be decompressed and played back in real time. And people were concerned about the performance of a Virtual Machine based language such as Java.

2011 My phone has a 1GHz Processor 512 MB RAM, 16GB of storage and can play MP3 and practically every other format.

In 15 years, my phone is more powerful than my computer was, not only is Java used for application development, but so is another interpreted platform (.NET). So yes given time, Moore's law will make any thing that is resource constrained today a viable technology.

Also, I've never heard of functional languages as being more memory intensive than OO.

6
  • @Mike Brown: thanks for your answer. Regarding my "hypothesis" about functional languages vs. memory consumption, this is due to the intensive use of immutable data structures - this would surely avoid lock and standard concurrency, but I guess it also translates in more memory usage.
    – rsenna
    Jan 19, 2011 at 19:52
  • 1
    (sorry sorry sorry) Java and .NET are not compiled (okay, going away, bye) Jan 19, 2011 at 20:16
  • 2
    @cbrandolino: Yes they are compiled. Java code is compiled to Java bytecode, which is once again JIT-compiled by the VM when it runs. Similarly, C#/VB/What-have-you are compiled to CLR bytecode, which is JIT-compiled by the CLR when it runs.
    – Anon.
    Jan 19, 2011 at 20:35
  • 1
    @anon: wha, sorry: I meant not interpreted - just woke up. Jan 19, 2011 at 20:37
  • 1
    An interesting sidenote is that Moore's law no longer translates into faster processors, but more processors on the same die. So Moore's law still holds but the dramatical increase of single-cpu's doesn't
    – Homde
    Jan 20, 2011 at 9:48
2

A huge problem today is that memory latency has hardly improved at all. This article estimates that over the last 20 years memory has improved

  • size - 128x
  • Bandwith - 20x
  • Latency - 1.3x

Processors combat this with larger and more advanced caching systems and pre-fetching. And as a result it becomes more important to keep memory access predictable and avoid random access.

Most of the benefit of functional programming is related to making it easier to parallelize computation, and this is clearly an increasing concern since processor frequency has stopped scaling.

I'm not aware of any significant difference regarding memory usage of functional programming. I would think that this is much more related to language design and compiler efficiency than programming paradigm. If performance is important, low level access to the hardware is useful, and functional languages typically does not provide this.

It seem to me that functional techniques has become much more popular, in large part due to increased productivity and easier parallelization. Pure functional languages has remained somewhat uncommon.

1
  • 1
    More than that: memory access is at least 10x slower than it was in the days of the 8080, in terms of processor clock cycles per memory access. Except for L1 cache, which corresponds more closely in both size and speed to what the 8080 called “RAM”. Jul 11, 2019 at 7:05
1

Moore's Law was originally about the number of circuit elements that would fit on a chip. The comparison to increased processor speed came later.

Faster and smaller elements benefit both the CPU and memory, so one would expect a roughly parallel increase in CPU and memory capability. Simply as a matter of qualitative impression, it seems to me that they've been increasing more or less together, along with hard disk size. (My first computer with a hard disk was a Mac SE, with an 8 MHz 68000 processor (16/32 bits), 1M of memory, and a 20M disk drive. Everything's increased in my current machines by factors between one and ten thousand.)

1

As others have stated Moore's law was really about transitor density. Since smaller geometries mean faster switching and data travel times, we saw speed improve along with density and the two were conflated. Several years ago a total power dissapation barrier was reached, faster clockrates are more power hungry, and without exotic cooling mechanisms we couldn't continue to advance them. But transistor budgets have gone on exponentially increasing, that means greater parallelism both within a core, and by replication of more cores has been pursued instead. Also caches become bigger with time, and this helps reduce the CPUspeed to memory speed mismatch penalty somewhat.

Functional programming, or any other technique which trades off programmer complexity for (somewhat) reduced resource usage efficiency is a consequence of the advances in HW capability. The sweetspot in the tradeoff between source code complexity and raw performance changes when machine capabilty increases. Also the capability to afford/contemplate more complex computations, leads to demand for more complex applications, which has an impact on the sweetspot of the tradeoff as well. If we only had kilohertz clocks and small memories, we would probably consider assembly to be the highest level language we would want to consider.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.