When Java was first designed it was considered appropriate to leave out anonymous functions. I can think of two reasons (but they might be different from the official ones):
- Java was designed as an object-oriented language without functions, so it was not very natural to have anonymous functions in a language without functions. Or at least, this would have influenced the design of the language a lot.
- Anonymous functions were not popular in the programmer communities that Java was meant to attract (C, C++, Pascal?). Even now, many Java programmers seem to consider these features quite exotic (but this will probably change very quickly with Java 8).
In the following years, as Robert Harvey has explained, the policy of Sun was always to keep Java backward compatible and very stable.
On the other hand, other competing languages have emerged (the most important being C#, which was born as a Java clone and then took its own development direction).
Competing languages have put Java under pressure for two reasons:
New features can make certain programming idioms easier to write, making the language more attractive for programmers. Normally the set of features provided by a language is a compromise between expressive power, language complexity, design coherence: adding more features makes a language more expressive but also more complex and difficult to master.
Anyway, in the last few years Java's competitors added lots of new features that Java did not have, and this can be considered an advantage.
Yes, unfortunately this is a factor in technology choice, at least from what I can see in my daily experience as a programmer: a tool must have a certain feature, even if most members of the team don't know how to use it and those who would be able to use it don't need it most of the times.
Hype can be even more important for non technical people like managers, who may be the ones who decide the platform for a certain project. Managers sometimes only remember some keywords like lambda, parallelism, multicore, functional programming, cloud computing, ... If our technology of choice has a green mark
on each item of the list, then we are up to date.
So IMO for some time Java has been caught between
- the original policy of language stability and design simplicity, a huge code base and developer community on one hand, and
- the pressure of competing languages that could attract Java programmers, C# at first, and then Scala, Clojure, F# (I name the ones I am aware of, there may be others).
Eventually Oracle decided to upgrade Java to make it more competitive.
In my opinion, the new features address especially Java programmers that might be tempted to switch to C# and who see other languages like Scala and Clojure as too different from Java.
On the other hand, developers who have some experience with functional programming and still want to use the JVM have probably already switched to
Scala, Clojure, or another language.
So the new Java 8 features will make Java more powerful as a language and the declared focus is concurrent and parallel programming, but the upgrade seems to address also the marketing aspects (Mark Reinhold, chief architect for Java at Oracle, said: "Some would say adding Lambda expressions is just to keep up with the cool kids, and there’s some truth in that, but the real reason is multicore processors; the best way to handle them is with Lambda", see this article).
So, yes, many (all) Java 8 features were well known already, but why and when a feature is added to a language depends on many factors: target audience,
existing community, existing code base, competitors, marketing, etc.
A short note regarding "... I had read about streams in SIC (1996).": do you mean that you need Java 8 lambdas to implement streams? Actually you can implement them using anonymous inner classes.