The networking data race condition from hell
I was writing a networking client/server (Windows XP/C#) to work with a similar application on a really old (Encore 32/77) workstation written by another developer.
What the application did essentially was share/manipulate certain data on the host to control the host process running the system with our fancy PC based multi-monitor touchscreen UI.
It did this with a 3 layered structure. The communications process read/wrote data to/from the host, did all of the necessary format conversions (endianness, floating point format, etc) and wrote/read the values to/from a database. The database acted as a data intermediary between the comms and touchscreen UIs. The touchscreen UI's app generated touch screen interfaces based on how many monitors were attached to the PC (it automatically detected this).
In the time frame given a packet of values between the host and our pc could only send 128 values max across the wire at a time with a max latency of ~110ms per round trip (UDP was used with a direct x-over ethernet connection between the computers). So, the number of variables allowed based on the variable number of attached touchscreens was under strict control. Also, the host (although having a pretty complex multi-processor architecture with shared memory bus used for real time computing) had about 1/100th the processing power of my cell phone so it was tasked to do as little processing as possible and it's server/client had to be written in assembly to assure this (the host was running a full real time simulation that couldn't be affected by our program).
The issue was. Some values, when changed on the touchscreen wouldn't take just the newly entered value but would cycle randomly between that value and the previous value. That and only on a few specific values on a few specific pages with a certain combination of pages ever exhibited the symptom. We almost missed the issue completely until we started running it through the initial customer acceptance process
To pin down the issue I picked one of the oscillating values:
- I checked the Touchscreen app, it was oscillating
- I checked the database, oscillating
- I checked the comms app, oscillating
Then I broke out wireshark and started manually decoding packet captures. Result:
- Not oscillating but the packets didn't look right, there was too much data.
I stepped through every detail of the comms code a hundred times finding no flaw/error.
Finally I started firing off emails to the other dev asking in detail how his end worked to see if there was something I was missing. Then I found it.
Apparently, when he sent data he didn't flush the array of data before transmission so, essentially, he was just overwriting the last buffer used with the new values overwriting the old, but the old values not overwritten still being transmitted.
So, if a value was at position 80 of the data array and the list of values requested changed to less than 80 but that same value was contained within the new list, then both values would exist in the data buffer for that specific buffer at any given time.
The value being read from the database depended on the time slice of when the UI was requesting the value.
The fix was painfully simple. Read in the number of items incoming on the data buffer (It was actually contained as part of the packet protocol) and don't read the buffer beyond that number of items.
Lessons learned:
Don't take modern computing power for granted. There was a time when computers didn't support ethernet and when flushing an array could be considered expensive. If you really want to see how far we've come, imagine a system that has virtually no form of dynamic memory allocation. IE, the executive process had to pre-allocate all of the memory for all of the programs in order and no program could grow beyond that boundary. IE, allocating more memory to a program without recompiling the whole system could cause a massive crash. I wonder if people will talk about the pre-garbage collection days in the same light someday.
When doing networking with custom protocols (or handling binary data representation in general) make sure you read the spec until you understand every function of every value being sent across the pipe. I mean, read it until your eyes hurt. People handle data by manipulating individual bits or bytes have very clever and efficient ways of doing things. Missing the tiniest detail could break the system.
The overall time to fix was 2-3 days with most of that time spent working on other things when I got to frustrated with this.
SideNote: The host computer in question didn't support ethernet by default. The card to drive it was custom made and retrofitted and the protocol stack virtually didn't exist. The developer I was working with was one hell of a programmer, he not only implemented a stripped down version of UDP and a mimimal fake ethernet stack (the processor wasn't powerful enough to handle a full ethernet stack) on the system for this project but he did it in less than a week. He had also been one of the original project team leaders who had designed and programmed the OS in the first place. Lets just say, anything he ever had to share about computers/programming/architecture no matter how long winded or how much I already new, I'd listen to every word. There is nothing more valuable than working with good people who have a genuine passion for what they do.
