As with most things, the useful heuristic is "how would I do this on the web?"
The first question to address is your resource model: are the full listing and the summary two different resources? or are they two different representations of the same resource?
In the case where the different representations are different resources, then they have different identifiers. You can use any convention you like for distinguishing the two identifiers. Using a query parameter is fine. Using a path parameter is fine. There are technical trade offs (HTML form support vs relative referencing) in addition to the aesthetics (which spelling looks nicer in the log file) but the machines don't care which spelling you decide on.
If they are two different representations of the same resource, then you are probably going to end up leaning into proactive negotiation, and helping general purpose caches do the right thing with the Vary header.
Some things to consider: if the rich representation and the summary representation have the same identifier, then it becomes that much more difficult to share links to a particular representation. The negotiation initially performed by a general purpose client is likely to include the same headers that it sends to every other resource on the web. "Please send me an HTML representation if you have one, in my human's preferred language". So which ever of the two representations you choose as the "default" is the one that people are going to end up with most of the time.
Also, it's probably worth noting that Fielding 2006 would likely discourage you from considering these two representations variations of the same resource
We encourage resource owners to only use true content negotiation (without redirects) when the only difference between formats is mechanical in nature.
Finally, note that REST doesn't constraint your implementation at all. It's perfectly reasonable to have two different resources implemented using the same controller, and it is of course in bounds to choose identifier spellings that make your implementation easier to maintain (ie: by conforming to the conventions of your server framework).
Then the other side of this is, using "?summary", I get a very different JSON structure than without.
Yup - perfectly reasonable as far as REST is concerned. You'll need some sort of metadata to identify the schema of the representation that is being used. That might end up being a hint in Content-Type header, or it might be communicated using Web Linking.