I'm having trouble planning the structure of my server side workflow and the technologies I should use.
The basic structure and tasks are:
Now, things to consider:
1.the server listens to multiple "folders" in the DB at once.
the server need to process many changes to the db - about 1000 per sec for all folders in the DB (not each folder, in total). The changes can come from many users, the server doesn't need to track (for now) where each change came from.
The process of getting the data related to the change can be costly - it involves searching large files for a certain expression within them. The data that gets stored in the Obj regarding the change is not that big, though. The size of the change Obj is not big - JSON Obj with about 20 short lines each, tops.
The order of creating the change Obj and sending it to the UI can be any order, because a time stamp is part of the data that is stored in the Obj, and I will make the UI sort everything (or maybe I should do it at the server side? where would you guys put this task?). Therefor I'm making the fetching changes and sending the Obj to the UI processes async.
UI and its communications with the node server needs to be compatible with Microsoft Explorer (version can be determined by me). External libraries can be used for achieving this, if necessary. The server will receive messages from the UI, like 'start' and 'stop', not continues ones.
What I want to avoid is:
slowing down the server due to listening to multiple folders and handling so many changes at once.
clogging the connection between the server and the UI by sending so many change Objs.
When I searched for info about this issue, I got across many useful technologies:
Websocket libraries - ws, socket.io etc.
Server event emitters.
Node packages and libraries for the "listen" functionality - fs, chokidar, etc.
So I was wondering:
What is the best structure and technologies to use for each part of the workflow in this case?
Should I use websockets? SEE? or plain HTTP REST API (with res.write in express)?
Would Node Clusters help with the parallel listening and fetching data whenever the folder changes?
What is the most efficient way to handle so many changes, and sending them to the UI?
I'm looking for advice regarding each step, for example:
Step 1: listening to multiple folders is most efficiently done using technology X, because...
Step 2: ....
Thanks a lot!