We have a number of IoT devices that communicate over TCP/IP to a server. Once connected, the server can then talk to the device, get data or send down commands.
The server accepts incoming connections from hundreds of IoT devices on a specific port.
My question is around testing of new code and deployment. Heres my scenario. There is a bug with one of the devices acting strangely or doing something wierd. Now normally you would try and replicated the bug in a test environment. But the thing is this is very hard to do. So what I end up doing is debugging code on the PRODUCTION server and step through the code for that particular device.
You see most of the bugs are hard to replicate in a test environment. You actually need to debug code on the production server connecting to a live device.
As you can imagine this is not ideal but if a particular device is playing up in the field, you want to be able step through the code with that device.
At first I thought I could divert the traffic of this particular IP to a test server with Visual Studio installed and then take my time to debug the problem. Only you cant divert a single IP. You can only port forward a whole port to another server. This means the whole estate of IoT devices get sent to the debug server.
I would like to hear how other people would go about this problem? Is there another way to single out an IP and debug it in isolation without affecting the other production devices. Or is there another way of approaching this problem all together. (By the way, these devices are geographically spread across the country which means we cant just go and change the IP on site)
Obviously this method plays havoc with change management as I am constantly putting debug code on the server!! - Instead of having the luxury of solving the problem on a test server I am messing with production!
Is there a better way of approaching this problem? Any thoughts would be very much welcome and thanks in advance.