27

This question already has an answer here:

I just learned about Scala yesterday, and I'd like to learn more about it. One thing that came to mind, however, from reading the Scala website is that if Scala runs on the JVM, then how is it possible for bytecode compiled from Scala source to be able to achieve things that Java can't readily do, such as (but not limited to) reified generics?

I understand that the compiler is what generates the bytecode; so as long as the compiler can massage source code into valid bytecode supported by the JVM, then both should be equivalent. But I was under the impression that Java couldn't even reify its own generics, so how could another compiler be able to pull this off?

marked as duplicate by gnat, Ixrec, user22815, GlenH7, user40980 Apr 27 '15 at 1:09

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 18
    If it's possible in Scala, then it's possible in Java and every other Turing complete language. The more interesting question is one of practicality rather than one of possibillity. – yannis Apr 25 '15 at 19:05
  • 5
    It is not a matter of what is possible, but what is easy. Think not about the JVM, bytecode, or language, but what the compiler does. – user22815 Apr 25 '15 at 19:22
  • 11
    VM Spec, 1.2 The Java Virtual Machine: "The Java virtual machine knows nothing of the Java programming language..." – gnat Apr 25 '15 at 20:11
  • 5
    @gnat Yeah, but it knows a decent amount about the Java object model. You couldn't implement C++ on it, and you probably couldn't implement C# either - at least not well. – Random832 Apr 26 '15 at 1:35
  • 2
    I'm not going to downvote but this question is senseless, as per the answer given by @FilipHaglund. Same question can be asked about any two languages running on the same hardware platform. In that respect there's no difference between a virtual machine like the JVM or an x86 or a Unix based machine. – Vector Apr 26 '15 at 3:27
29

"All" programming languages run on x86, so how can they be much different from each other?

Brainfuck and Haskell are both Turing complete, so they can both do the exact same tasks.

There's a bit of room for syntax changes, syntax sugar and compiler magic in between. You can do quite a lot in there, but there is always a limit. In your case, it's JVM byte code.

If Java can produce the same byte code as Scala, they are equivalent. It could, however, be the case that a new feature in the JVM gets implemented in only Scala. Very unlikely, but possible.

  • 12
    The JVM already has features that are intended to be implemented by languages other than Java. See: JVM Support for Non-Java Languages – kapex Apr 26 '15 at 13:42
  • 6
    I don't think every (valid) bytecode sequence can be produced from Java. Nor probably is this the case for scala. So this argument that the two languages are equivalent is not valid (although probably the are equivalent nevertheless) – Marc van Leeuwen Apr 26 '15 at 13:44
  • @Marc Absolutely. For example goto (the bytecode) allows you to jump to any valid instruction offset in the same method, but javac would not generate code that jumps into the middle of a loop from outside since you can't express such a thing in Java. Some JITs exploit such properties to generate better code (not sure if HotSpot does too, I'd guess it'd need to fall back on interpreting methods that violate this #property). – Voo Apr 26 '15 at 16:40
21

To address the specific issue that you raise, of reified generics . . .

In many contexts, type parameters are actually saved in class-files and exploitable via reflection, even despite erasure. For example, the following program prints class java.lang.String:

import java.lang.reflect.Field;
import java.lang.reflect.ParameterizedType;
import java.util.ArrayList;

public class ErasureWhatErasure {
    private final ArrayList<String> foo = null;

    public static void main(final String... args) throws Exception {
        final Field fooField = ErasureWhatErasure.class.getDeclaredField("foo");
        final ParameterizedType fooFieldType =
            (ParameterizedType) fooField.getGenericType();
        System.out.println(fooFieldType.getActualTypeArguments()[0]);
    }
}

All "erasure" means is that when you create an instance of a parameterized type, the type parameter is not recorded as part of the instance; new ArrayList<String>() and new ArrayList<Integer>() create identical instances. But even in Java, there are a few well-known workarounds, such as:

  1. Something like new ArrayList<String>() { } actually creates a new instance of an anonymous subclass of ArrayList<String>, and the subclassing relationship does record the type parameter. So you can retrieve the String reflectively. (Specifically: ((ParameterizedType) new ArrayList<String>() { }.getClass().getGenericSuperclass()).getActualTypeArguments()[0] is String.class.) This is exploited by various frameworks, such as Guice and Gson.
  2. If you're creating your own class, you can require the type parameter to be passed via the constructor, and store it in a field. (The compiler can help you enforce that the constructor argument matches the type parameter. And if the type argument is itself generic, you can combine this with workaround #1 to ensure that you have the whole type argument.)

Another language residing on the JVM could reify generics by making #2 happen implicitly; whenever you declared a generic class, the type-parameter field(s) and constructor-parameter(s) would be added implicitly, and whenever you instantiate or extend it, type-argument(s) would be implicitly copied to constructor-argument(s) to populate them. Java already does the same thing — implicit fields and constructor-parameters/arguments — in a different context, namely, for local classes that refer to final local variables in their containing methods.

The main limitation is that generic classes in the JDK — the collections framework, java.util.Class, etc. — don't already have this setup, and alternative languages running in the JVM can't add it to them. So such languages would need to provide their own equivalents of the JDK classes. But they can still use the JVM itself.

7

JVM bytecode pretends to be a kind of generic machine code, and it is indeed, so... what makes you think it couldn't support any other language? JVM bytecode is a Turing complete language, and thus, every program, no matter in which language is written, can be compiled/translated into bytecode.

There are a lot of languages which already have a bytecode compiler (e.g. Jython for Python and JRuby for Ruby) and there are also quite different from Java.

Note that technically, every Turing complete programming language can be compiled into another. It could be possible to compile JS to C or Ruby to Python, for example.

  • My thought was more that since Java can't handle reified generics, that any bytecode read by the JVM would also have that limitation. I hadn't considered that the compiler is really what controls that sort of thing. – agent154 Apr 25 '15 at 23:02
  • 6
    Turing completeness is a red herring though. Real cross-compilers (and the interpreter equivalent, whatever you want to call it) don't just need semantic equivalence, they also want reasonable performance and implementation effort and interoperability. For example, despite all their superficial similarities, properly compiling Python to JS amounts to re-creating a whole Python implementation in JS, and that is only barely practical because emscripten already exists. And that doesn't even give you a fraction of the Python-JS interoperability Scala and Java have. – user7043 Apr 26 '15 at 0:24
  • I was saying that it is theoretically possible, not that it is practical or sensible. It was only an example to picture that an algorithm written in a Turing complete language has always an equivalent algorithm in every other Turing complete language, and thus, they are translatable the one into the other. That's why we have emulators, it's the same principle. A Turing Machime could emulate an x86 (and everything else in fact), but yeah, it would be really slow, costly and pretty pointless. – Eneko Apr 27 '15 at 18:05
3

In a nutshell, Scala can do it because the Scala compiler is a master in code transformation/generation. Which means Java could do it too (maybe it does already). The trick is not made at the bytecode level but at the source level.

Could you guess the output of this code:

def test[T](f : => Any) : T = {
  try { val x = f.asInstanceOf[T]
    println("f.asInstanceOf did not raise an error")
      x
  }
  catch { case e : Throwable =>
    println("f.asInstanceOf did raise an error")
    throw e
  }
}

val x = test[Int]("x")

(Not so) Suprisingly it outputs

f.asInstanceOf did not raise an error
java.lang.ClassCastException: java.lang.String cannot be cast to java.lang.Integer at scala.runtime.BoxesRunTime.unboxToInt(BoxesRunTime.java:105)

... 33 elided

Because of type erasure, f.asInstanceOf[T] can not fail but obviouly a String is not an Int so it has to fail somewhere. Fortantely Scala brings solutions to this problem:

import shapeless.Typeable
import shapeless.syntax.typeable._

def test[T : Typeable](f : => Any) : T = {

    f.cast[T] match {
        case Some(x) => {
            println("f.cast[T] succeed")
            x
        }
        case None    => {
            println("f.cast[T] failed")
            throw new RuntimeException("cast failed!")
        }
    }
}

val x = test[Int]("x")

The main difference here is we declare T as Typeable. Scala have several ways to provide runtime representations of types.

2

Reified generics do not require JVM support. Yes, they would be easier and most performant with JVM support, but JVM support is not necessary. For instance, a Scala compiler could, for every class featuring a type variable, add a field that stores the corresponding object:

class List<T> {

}

void test() {
    List<?> list = new List<String>();
    List<Integer> intList = (List<Integer>) list;
}

would be compiled to

class List<T> {
    final Class<T> tClass;
    public List(Class<T> tClass) {
        this.tClass = tClass;
    }

    public <O> List<O> castTo(Class<O> oClass) {
        if (tClass == oClass) {
            return (List<O>) this;
        } else {
            throw new ClassCastException("Incompatible type parameter: " + tClass);
        }
    }
}

void test() {
    List<?> list = new List<String>(String.class);
    List<Integer> intList = list.castTo(Integer.class);
}

There are of course more elaborate translation strategies that more seamlessly integrate with the host type system, for instance by actually having separate classes for different type arguments to the same generic type:

class List<T> {

}

class List#Integer extends List<Integer> { }

class List#String extends List<String> { }

void test() {
    List<?> list = new List#String();
    List<Integer> intList = (List#Integer) list;
}

(There would of course be some challenges not apparent in this trivial example, for instance recursive type arguments, proper isolation of different class loaders, ...)

The reason Java does not have reified generics is not that reification would be impossible on the JVM, but that reified generics would have broken backwards compatibility (in particular binary compatibility) with existing Java programs - something Scala did not have to worry about.

  • I don't think the issue was "broken compatibility" so much as "complicated interoperabilty". If code has an object Producer which constructs and returns a reference to an implementation of List, all of whose members are of type Animal, and another object Consumer which expects to be given a type-erased List<Animal>, the code can easily use the two objects together. If, however, Producer had expected a reified TList<T>, it would have been much harder to have an object which use Producer to supply things for Consumer. – supercat Apr 27 '15 at 1:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.