Instead of using the same API key used to send the error logs, use one API key per user. In other words, a person purchasing your library will have to generate his own API key, and then give it to the library he purchased. This way, the person will be identified, and the level of identification depends on your specific needs (it could be a simple e-mail address or much more complete identification process).
This way, you'll avoid most of trouble coming from your customers. Someone who knows he's identified would be much less inclined to DOS or spam your server, because he would believe that legal action from your side will be easier, as well as because he would know that his API key can be blacklisted.
This is all but easy to develop, so make sure it's really worth it. The benefit from doing it this way is to save your bandwidth, and the bandwidth can indeed be very costly.
If bandwidth is not an issue, just let all the error messages in. Then, you'll be able to filter them based on the IP address, the number of messages in a given time, the messages themselves or their patterns, etc. After all:
If you're low scale (i.e. if you receive a few dozens of error messages per day), filtering spam manually shouldn't be a problem.
If you're large scale, receiving thousands of messages per minute, you can leverage the scale in order to identify the abusers and automatically get rid of them.