benchmarks for quasi real-time software

Question

I'm wondering if there is any benchmarks which can help to understand real-time characteristics (response time) of non-real-time software. For instance C# application on Windows or MSSQL Server.

I know that Windows is not real-time OS, however it does not prevent us to use it in Real-time system if there some additional real-time layer.

Let's say we have some real-time system, which reads and store data from some ADC. Data are read from ADC and timestamped on FPGA w/ some real-time Firmware. That part of the system is responsible for taking samples from ADC with some precise interval (let's w/ 10kHz frequency) and for timestamping the data. Afterwards firmware pack samples into data package (let's say 0.1 sec) transfer data via TCP/IP to application on Windows machine. Windows machine should store the data and display it. Yes, application on windows is not real-time as well as TCP/IP connection does not guarantee data delivery in fixed time. For instance

• in some case data transfer might take 0.01sec and 1 sec in another case.
• Windows may interrupt application's threads for a few second.
• application might interrupt itself when garbage collector is working

so as a result we may have a few second interrupt in the part of the system which is beyond Controller/Firmware. I.e. response time from non real-time subsystem is varying in some range. However this issue could be solved if there is buffer big enough on controller, so it can accumulate data waiting for response from non real time subsystem.

In order to be sure that buffer on real-time subsystem is big enough to support non-real-time subsystem we need to understand response time probability distribution. Something like:

• response time <1sec in 80% cases
• 1sec
• 2sec
• 5sec
• 10sec

That is basically the question. Any benchmarks for Windows, C# or MS SQL? Any methodic?

TPC does nice benchmarks for SQL Servers performance however they total performance, and do not consider response time AFAIK.

Answer 1

Having done a lot of performance testing, I would suggest that you do not look for existing benchmarks as there are too many variables.

My suggestion is to establish your own benchmarks with your software (and bear in mind that if you install any software on the windows machine it can have a significant impact, particularly those that change services or background tasks - including windows updates - so try and keep your configuration unchanged once you have established your benchmark).

To establish your benchmark build your controller software with as large a buffer as you can afford for a test, make sure that you have a means of recording buffer use (min, max, average, response times, time in buffer, whatever indicator will help you fine tune). Design a performance test that will push your system to the limits (you may need a 24 hour test if your systems are meant to run h24; if the amount of data changes with time of day, choose a representative load for the busiest day and extrapolate by 10 or 20%; if data rate is constant, then it's easy, just do that). Chart your controller and client buffer, request, response & timing statistics. That should give you all the ammunition you need to tune your controller.

Lastly Windows has a number of network settings that can be changed that will affect network performance. This is just one set of variables that mean standard benchmarks won't help, if any exist out there.