Suppose there's a GPLv3 app that has a GUI and a CLI. Can I legally expand its CLI (to give me access to functionality hitherto only available via the app's API), release the result under GPL and then create a non-GPL (perhaps proprietary) app that depends on this modified GPL app (via the expanded CLI)?
There are a couple of exception paths that might enable you to do what you're suggesting.
The first approach is to have the GPL'd app only provide data to the second non-GPL app. See here for additional details.
Another approach relies upon creating a plugin structure, but this is a tricky approach. See here for FSF's comments on this approach.
Depending upon the original GPL program, you might be able to write a module with additional permissions, but this is likely going to be tricky to do as well. See this related FSF FAQ
Finally, if you never distribute your second application then you are not required to distribute the source as per the terms of the GPL.
As a simple test answer this question: Is my proprietary program useful without the GPL component? Ignore how the components communicate for now.
If the answer to this question is no, then you probably should consider your program a derivative work and thus must be covered by the GPL.
If the answer is yes, then you can probably distribute your proprietary program with the GPL program. But according to the FSF GPL FAQ:
To do this validly, you must make sure that the free and non-free programs communicate at arms length, that they are not combined in a way that would make them effectively a single program.