Don't bother trying to identify a hacked client
I'm wondering the following: how do I prevent this without rewriting the entire server to be authoritative which would be a huge uplift?
Unfortunately, you can't.
- Your game state is vulnerable to exploits because the server doesn't protect / manage it.
- Your server is "vulnerable" because it broadcasts exploited state changes to its clients.
The only way to fix that is to have the server manage the game state.
... That is: I know to an extent it's not entirely preventable, but I'm wondering if we did something like had a private/public key exchange before login to the server, and those keys were gitignored, or something to that effect: could we do it?
That key exchange could probably be easily recovered if somebody wanted to go through the trouble of making a hacked client. The hacked client would just be modified to send data as if it was the real one.
Pitfalls of Kicking / Banning hacked clients
Previous answers have mentioned that you can simply kick/ban hackers, but this is a bit more complex than it might seem at face value.
The success of this depends on:
1. How well the community can identify when somebody is hacking
If you see a player teleporting all over the map, sure, it's rather obvious, but can you detect more subtle problems (e.g., Like hacked clients being able to see parts of the map they shouldn't be able to see? Or aimbots?)
If your game is competitive, players may argue over who was hacking, or who just had good aim / good reaction time / good luck, etc.
If your engine is so vulnerable that clients decide kills / damage, your moderation team will likely have a lot of work cut out for them. Any improvement you make to your game's security will mean less manual work reviewing alleged hackers' gameplay videos, etc.
2. How well the server can identify your players
If you have a large community, attempting to protect all public facing servers where anyone can join will be rather difficult.
Identifying users by IP address
Advantages:
- Ease of use. No additional signup needed for your game, most users will be able to just download it and play.
Disadvantages
If your servers just accept data from any IP address,
- Users can easily use proxies to get around IP bans.
- Legitimate users with consumer internet connections will change IP addresses frequently.
- From my experience some server admins' attempts to block known hackers with ban ranges inadvertently caused 80% of an ISP's customers to get blocked.
- New users might try to play for the first time and find out that they are banned for no reason, I'd recommend giving them a way to contact somebody about this.
Other identification methods
A game engine I used got around the IP address problem by having users sign up for accounts, and then allowing servers to whitelist only certain players. These accounts were only required for more "serious" servers (for tournaments, etc).
On a smaller scale just allowing your servers to be password protected may go a long way towards keeping hackers out of competitive, serious games.
Protecting the game state is the best way to keep hackers away
The best way to reduce the risk of a hacked client ruining the game for other clients is to make it so that every change to the game state (that affects other clients) is validated by the server.
This way even if a hacked client decides to ignore the server's updates saying that a player took damage, all it does is desync that player's own game.
Design challenges of making a less vulnerable server
There's a lot to cover, but I'll mention the main issues I've faced as somebody who worked on this:
Making a responsive client
In order for the game to remain fluid, the client will likely need to run some code on its own and make some guesses on what the next valid frame will be.
Some operations that the client does, like mouse movements for example, happen far too fast for the server to validate, though if you don't do this, you may open yourself up to hacked clients that move the mouse impossibly fast or accurately.
There may not be an easy answer for all possible exploits, for example aimbots and wall hacks are still being made for even well funded and popular FPS games.
Allowing the client to receive data from the server
Knowing what data on the client to throw away when the server sends the client an update is not easy.
The client's code may not only have to update the main game's data but also any interpolations that the client was in the middle of, to avoid distracting jittering / rendering glitches.
