I'm having trouble planning the structure of my server side workflow and the technologies I should use.

The basic structure and tasks are:

Node server workFlow

Now, things to consider:

1.the server listens to multiple "folders" in the DB at once.

  1. the server need to process many changes to the db - about 1000 per sec for all folders in the DB (not each folder, in total). The changes can come from many users, the server doesn't need to track (for now) where each change came from.

  2. The process of getting the data related to the change can be costly - it involves searching large files for a certain expression within them. The data that gets stored in the Obj regarding the change is not that big, though. The size of the change Obj is not big - JSON Obj with about 20 short lines each, tops.

  3. The order of creating the change Obj and sending it to the UI can be any order, because a time stamp is part of the data that is stored in the Obj, and I will make the UI sort everything (or maybe I should do it at the server side? where would you guys put this task?). Therefor I'm making the fetching changes and sending the Obj to the UI processes async.

  4. UI and its communications with the node server needs to be compatible with Microsoft Explorer (version can be determined by me). External libraries can be used for achieving this, if necessary. The server will receive messages from the UI, like 'start' and 'stop', not continues ones.

What I want to avoid is:

  1. slowing down the server due to listening to multiple folders and handling so many changes at once.

  2. clogging the connection between the server and the UI by sending so many change Objs.

When I searched for info about this issue, I got across many useful technologies:

  1. Websocket libraries - ws, socket.io etc.

  2. Node Clusters.

  3. Server event emitters.

  4. Node packages and libraries for the "listen" functionality - fs, chokidar, etc.

So I was wondering:

What is the best structure and technologies to use for each part of the workflow in this case?

Should I use websockets? SEE? or plain HTTP REST API (with res.write in express)?

Would Node Clusters help with the parallel listening and fetching data whenever the folder changes?

What is the most efficient way to handle so many changes, and sending them to the UI?

I'm looking for advice regarding each step, for example:

Step 1: listening to multiple folders is most efficiently done using technology X, because...

Step 2: ....

Thanks a lot!


  • 1
    how exactly does the user change the db? – Ewan Oct 19 '17 at 17:39
  • 1
    what database are you using? – Ewan Oct 19 '17 at 17:40
  • @Ewan It doesn't matter. The thing is that the server looks at folders that contain text files. its not DB per se, but some txt files that store info about mongoDB that work in the background. Everytime the mongoDB collection is changed, a third party program edits the txt files. Whenever one of the txt file is changed, I want the node server to report about the change to the UI – mizenetofa1989 Oct 19 '17 at 18:01

OK, so. The best way to solve this kind of problem is to catch the initial user action and make it fire off your process rather than monitor the effects of the existing process and try to infer what happened.

But you haven't give us any information about what that action is, what tech its using, whether you can alter it etc. We will have to look at other ways of getting a "hook" into the system.

Lets assume its a piece of third party software over which you have no control and which offers no APIs or hooks.

Next up, we have the database. Again its unclear quite how this fits into the picture. But its MongoDB. which unfortunately doesn't have Triggers. If it was MSSQL you could possibly add triggers to your tables and get them to execute whatever custom process.

Now there are apparently some tricks you can use. see this question https://stackoverflow.com/questions/9691316/how-to-listen-for-changes-to-a-mongodb-collection

Lets go on though and assume that those wont solve your problem. You then come to these 'multiple "folders" in the DB' lets assume these are txt files on linux.

There are some tools that can trigger command off file system events: http://inotify.aiken.cz/?section=incron&page=about&lang=en

If you set this up to monitor the files that will give you a hook.

Once you have your Hook, I would recommend using it to create a message on a queuing system such as rabbitMQ. This will enable you to run all your other programs and events off the queuing system rather than having to worry about how the event got to it in the first place.

Lastly, how do you "Push" the event to a webbrowser client? Here as you have already found out, websockets provides a solution and your queuing system will help with scaling the events over multiple webservers.

.... I'm going to go back to the start though and say, although I have seen systems that monitor files and do stuff when they change. They have always been fragile and effectively held together with string. You can probably cobble something together, but if you are "planning the structure of my server side workflow and the technologies" then it would be best to go right back and capture the initial user action that triggers the change.

  • Thank you for the answer! a few points: I presented the server as it is because this state is a given. Because of technical issues and the fact that I join the team at a late stage in prudoction, there is no way to directly listen to the changes in the DB, only the text files (its also not efficient, because the structure is a bit more complicated that what I presented here). The thing I want to focus on is how to handle with node many changes with a single or mutiple listen functions, and sending them ti the UI, which runs in browser in a different machine - all of that should run quickly – mizenetofa1989 Oct 20 '17 at 13:47
  • Well, like I say, I would go with the incron listener -> RabbitMQ -> node server(s) -> websocket – Ewan Oct 20 '17 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.