How to manage large scale project in node.js keeping everything Asynchronous?

Question

I have a large module which have to process more than 10k request/ response per second. Every request has a json and need to process it and verify it on the database and generate the response on the basis of the db query. Here is an example

 Function 1

 function  ('/get' , fucntion (req1 , res1 , next ){

  //to process the req data 
  processData(req1 , res1 );

 });


 Function 2

     processData(req1 , res1 ){

      var J = JSON.parse (req1.body)
     //it has to read db three times 
     //process the json build the response and return 

           condtionCheck(J , res1){

                    var a = someVlaue(){
                    //select a value from a collection set of nosql which have 
                    //more than 1000 document and 
                    //i have to itrate a for loop to check the condition .....etc 
                    //...........

                      }
                    ........

                      dataRead(var a , res1){
             // is it possible to send response for the req1 object here
                         res1.send({value b: abcd123})
                    }

              }

      }

     Function 3 ..... and so on 

The major problem is all the code i have written inside the process data is synchronous. Bacaue each code depends upon the previous call back and there are so many condition check are used in several times.

So It is good to put such a large processing in synchronous way inside node ? If i write the code using async some times all scenario got in a deadlock condition How to avoid such behavior? Does async or function like step have affect on performance ? On such a series of function how can we reduce latency ?

Answer 1

Short answer:

No, synchronous processing in node.js WILL cause performance problems.

Long answer:

You want to avoid synchronous processing in node as much as possible. Obviously, sometimes you're going to need it, but that may be a point at which you move the relevant synchronous code out of node and into something more suitable.

Basically, as node follows an evented model of concurrency, where a single thread runs a stack of events one at a time, any long-running events are going to block the execution of anything else. So, what you'll be looking to do is to scythe off as much functionality as possible into small, asynchronous, function calls.

What you'll want to do is look to use a library, such as async or Step, which allows you to avoid the "callback hell" approach to node programming. JQuery has a similar thing in the browser that you may have used, which is its promise() method and Deferred objects. This will make it much quicker to write, and maintain, your node source.

If you really do need to put a large synchronous procedure in your web app, then node is not the right tool for that job. Try looking at having another API that node can call out to when it needs to accomplish a long-running task, whether that's a command-line C++ app or a web API in a more standard-concurrency model language.