The specification is very clear about it
The Java Virtual Machine has a heap that is shared among all Java Virtual Machine threads. The heap is the run-time data area from which memory for all class instances and arrays is allocated.
The heap is created on virtual machine start-up. Heap storage for objects is reclaimed by an automatic storage management system (known as a garbage collector); objects are never explicitly deallocated. The Java Virtual Machine assumes no particular type of automatic storage management system, and the storage management technique may be chosen according to the implementor's system requirements. The heap may be of a fixed size or may be expanded as required by the computation and may be contracted if a larger heap becomes unnecessary. The memory for the heap does not need to be contiguous.
A Java Virtual Machine implementation may provide the programmer or the user control over the initial size of the heap, as well as, if the heap can be dynamically expanded or contracted, control over the maximum and minimum heap size.
The following exceptional condition is associated with the heap:
If a computation requires more heap than can be made available by the automatic storage management system, the Java Virtual Machine throws an OutOfMemoryError.
To answer your question:
When we say "the heap" in the context of the JVM, are we talking about somewhere inside the VM itself (which may or may not be allocated on the heap of the underlying physical computer)? Or do we refer to the heap of the physical computer?
When we speak of heap, we speak of already allocated memory for the JVM. So, it is inside (?) the JVM (if that makes any sense to say). The JVM has its own memory management.