I will admit that this answer is mostly speculation. First off, I've not heard of any package manager using something like NameCoin or any market approach to naming conflicts.
Frankly, I've never really heard of this as a problem in practice, though my experience is primarily with the Haskell Hackage package repository. In that community, occasionally there has been frustration expressed about some package taking a "good" name, but the newer package simply used a different name. I'm pretty confident no one has ever "squatted" on a name (i.e. uploaded a useless package just to tie up a name) in Hackage. If that ever did happen, the system admins would intervene in a "moderator" capacity.
Since it's extremely unlikely that you'd get different packages with similar (or the same) name and largely overlapping APIs (particularly in a statically typed language like Haskell), there's very little chance that you will accidentally choose the wrong package and not realize it... unless someone's being malicious.
For Hackage, I believe I would have heard about it if this had happened (and people realized). In fact, for the first few years anyone with a Hackage account could overwrite any other package with anything with no notice. There was no cryptographic signing or anything like that. (The security story now is much better.) That said, there were no real issues. Certainly no malicious behavior. (Admittedly, the account creation process at the time was something like you got an account if Ross Paterson thought you were a real person.) There is no clear mechanism on the Hackage site for dealing with a malicious package, no "report this package" link or even an email address suggested for that purpose. The most obvious thing to do if you saw a malicious package would be to email the admins through one of the general emails. In fact, even simply posting a message about it to one of the relevant mailing lists (including the main "announcements" list) would likely produce a prompt response from the admins.
Admittedly, the Haskell package repository is not nearly as high value a target or as high volume as the package repositories of more popular languages. Nevertheless, as a semi-random selection, none of Hackage, NPM, Maven, or CPAN have a simple, obvious method to "report" a package for any reason. Ruby Gems does have a "Report Abuse" link on the right, which apparently starts a new "discussion" thread on the help site. As far as I can tell, people behaving maliciously in the manners you've suggested just hasn't been enough of a problem to warrant any significant effort. Most language package repositories are centralized and so there is a clear "admin" who can be responsible for dealing with such problems as they are reported. I'm pretty sure in general the repository admin is treated as the de facto moderator, and more formal processes have generally not been found to be necessary. That said, any package that is reported is likely to be outright removed and the relevant user accounts potentially disabled probably with appeal only after the fact.
