Yes, you're right. But the stack basedness is just a part of the whole story. For example, the Java bytecode interpreter is stack-based as well (the compiled code works -- for efficiency reasons -- differently). This tells us, that any language can be transformed into a stack language.
What matters are the objects outside of the stack, those who can outlive the current method execution. As long as the language has nothing like
new, there are no such objects and you need no
delete nor GC.
A language lacking dynamic memory allocation is quite limited in its usefulness.