I hope this is not too opinionated because I'm looking for a finite answer.
I'm developing a UDP server in Python 3.x that is utilizing asyncio. The server is going to work with a game engine to process pretty much every interaction a player makes in the game. Therefore, I have to authenticate the game client with the game server in some way as well ensure replay attacks do not happen on top of everything else that could harm or spoof the game server.
When it comes to authenticating with UDP, I'm at a loss. My plan is to have both the game client and game server authenticate per user and game session. That means having something like a public key on the client end and a private key on the server end where the server can authenticate the client is approved.
During that authentication, I am going to generate a symmetric key that the game server makes and passes it down to the game client so every packet sent can be verified with that symmetric key using HMAC. If not, all packets are dropped.
My Question
Is this the best approach? Creating some type of public key where something like a token is generated per session to ensure packets coming to the UDP server are from authenticated clients? My worry here is the keys are still stored in a Windows EXE file and still likely can be cracked and extracted or am I just too paranoid?
Going to answer my own question!
No It's Not The Best Approach
The reason this is not the best approach is because even with this process, my use of public/private key was incorrect. I was not referring to encrypting data between the client and server, I was referring to creating a nonce with having some type of password on the client that matches the password on the server.
This means the moment I send credentials from the client to the server, it's being sent without encryption on the line or in the buffer to the server. Once this authentication is accepted by the server, it will hash the password, store it and then send the nonce back to the client, which will also be without encryption.
The username, the password, and the nonce will be vulnerable to Man in the Middle attacks from both a third-party and the client itself. Which means they can sniff the packets, snag the transmission and crack the user or attempt replays being they have the nonce.
Solution
My game client does not have SSL/TSL/DTSL options. This is why I actually posted the question, but did not clarify this. My server, as it's custom within Python, does have that ability though. But, I found a plugin for the game engine last night that allows for AES encryption of the data being written to the buffer before said buffer is sent as a packet to the UDP server, which means I can encrypt the passing of data in those packets in a secure way that hopefully the server can decrypt with a legit private key.
Doing this will be a good option because it likely will be my only option. Now when a malicious user sniffs the traffic, they wont be able to see credentials or the nonce. The only issue after that is cracking the EXE, which I'm fine with for now.
What is the best way to secure UDP server with HMAC?