You're mostly on the right track. The stack's purpose in general is to hang onto data you'll need later. Most of the time, that's done when calling subroutines, at the very least to save the return address, but also to save any state that may be destroyed otherwise and as a mechanism to store local variables and to pass parameters. In most multitasking systems, every execution context has its own stack for those purposes.
Many systems also use the context's stack when switching between them.
When a context stops executing, everything it had going on needs to be preserved. The stack will already hold most of that, but the state of the processor's registers has to be collected and stored as well.
This information could be stored in a data structure allocated specifically to do that job, maybe as part of a process control block. The thing about that approach is that you'd need one per context, and the structures would be occupying space unnecessarily while the context is running.
As it happens, there's already a block of general-purpose scratchpad memory available: the stack. Once control has been wrested from the context, there is zero chance that it will be pushing anything else there. This makes it safe for the supervisor to push the processor state there, and all it has to remember for later is the context's stack pointer. Because the context won't run until the supervisor gives it the okay, the last things pushed to the stack will always have been the processor state. When the context is to be restarted, the supervisor restores the saved stack pointer, pops the remainder of the processor state from the stack, jumps to the address where the program counter was when the context was interrupted (also stored on the stack). Then the context continues running as if nothing had happened.