1

I have some network device(IOT) that contain temperature sensors to collect the temperature of milk mixing tanks and send that temperature to another networked computer.I have more than 100000 of tanks to collect data.Data collecting interval is 5 seconds.It's requirement and cant change the 5 second interval.

Then any time withing one second they give at most 5*100000 temperature values and I need to send that data to server (MSSQL) and It's have another software get the data from DB and control the temperature of tanks.

At the moment I used synchronous TCP/IP Socket. When socket programing they open arbitrary ports.Then slow down the system. Any solution for this type of scenario ?

1
  • The idea of control software getting it's real-time data from database sounds like madness to me. And if you have throughput problems, then only solution is to profile your code and find the bottleneck.
    – Euphoric
    Jun 28, 2016 at 13:11

2 Answers 2

3

It sounds like your database comes in too early in the process.

I would go with a light queuing solution maybe ZeroMQ? (not sure what you can run on your devices)

the key thing I think is for the 'another software' respond directly to the message before putting it in the DB.

That would reduce your database load to just inserts and allow you to scale up the number of 'another software' Workers listening to the queue to cope with load AND/OR split the sensors up so they report to multiple queues/servers

architecture

1
  • I'll try and let you know the status.
    – Elshan
    Jun 28, 2016 at 18:26
0

So, just to be clear:

every 5 seconds a tank will send its temperature to a server that stores it in a DB and then another server program will read this value and send a message to the tank to regulate its temperature?

If so, the solution is quite simple: send the regulation command directly to the other IoT device that controls temperature, batch the temp data and send it in larger blocks to the DB server for reporting purposes.

Otherwise, if you are trying to maintain a straight throughput of data to the server, use a UDP socket that sends data to a socket without all the TCP overhead. You may find storing the data received in memory or a flat file per tank is easier to manage than a SQL Server DB that receives that many updates.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.