I'm currently developing a real-time application using .NET Core 2.2. Many IOT devices (could be up to 100) are concurrently sending messages via MQTT. These messages are parsed, elaborated to extract and validate the current position of each device based on some business rules and then persisted to a SQL Server DB using Entity Framework, so they can be displayed in real-time by clients connected to the server via SignalR.
What I'm experiencing even with few devices is that the actual elaboration of each message is nearly immediate (1ms long) and the persistence layer is taking up to 300ms at the point that the system is not able to catch up with the current flow of messages and positions are always delayed in the map display.
What can I do to improve this design?
What I tried:
- using a memory representation of the whole system and then saving it every few seconds. EF's DbContext is not thread safe so it doesn't like this approach. Using locks is even slower.
- caching data only mitigates the problem because most of the data is constantly changing and can't be cached.
- using a message queue?
- using MongoDb?