One problem I have with compiler optimization is, that I actually never know what the compiler (or in this particular case the Jit) does with my code. So is there any possibility for me to know whether a particular compiler optimization took place or not? I just want to be sure about that some objects (math vectors) are created on the stack and not on the heap.
See related Q&A here which suggests you can download a debug JDK and use command line options: -XX:+UnlockDiagnosticVMOptions -XX:+PrintEscapeAnalysis -XX:+PrintEliminateAllocations
to print out the escape analysis events as they happen.
The short answer is that you cannot.
Whether or not your vectors are created on a stack or on the heap is not determined by JIT, it is a property of the JVM that is running your application.
The JVM specification has sections describing the stacks and the heap. The section on stacks says
Because the Java virtual machine stack is never manipulated directly except to push and pop frames, frames may be heap allocated.
With a traditional compiler you can usually look at the assembly language it emits and determine whether or not certain optimizations have taken place (not entirely, as multiple optimizations may wind up producing the same code, but what you're looking for is the effect, not the cause). For JITs, not so much, unless you've got a "chatty" one that emits a log or trace message when it performs its optimizations. I know that with some JVMs you can see whether specific methods have been JITted, but not when or why (although often the criteria the JIT uses are documented).