TL;DR: How distributed open-source apps like Scuttlebutt are secured from DoS and hackers who can make custom version of application?
I'm struggle with designing an open-source distributed application architecture. I want to create an application consisting of open source server, client, and provider. Client sends requests to one random provider instance that have list of all instances both client and server, and sends it to one of the random server, which, after processing the information, sends a result request back to the client. Every part of this distributed app is open-source, so everyone can create their own instance of client, provider, server, and everything seems to be fine, but what if some programmers will have bad intentions, and they will change client code in the way, that it will send millions of requests (DoS attack) to the specific, not random provider, or change providers with, so it will send all requests to one specific server? Also they can change server code, so if client expects to get a specific picture from server database, hacker will send some inappropriate pictures to all avaible clients.
If I hardcode some kind of verification, like hashing of important functions of API, then hacker will just remove this in his own fork. Therefore, I cannot solve this problem in any way, except by making the code of one of the parts private. For example I can make provider application with private code, so it will check hash of both client and server, and if this check fails - provider will delete this instance from list of instances. This solution sounds good, but in this case, the whole project will no longer be open source.
Summarize: I want to create an open source distributed application, so everyone can make their own instance, improve it, add new functionality, but how can I secure it, so this ability to create custom versions should not be misused for DoS, sniffing, or information corruption in conjunction with all many different versions working together.
I don't quite familiar with this topic, so I'll be glad if you can give me advice, a link to an article on a similar topic, or a book.