I am running a cron job every 5 mins to get data from 3rd party API, It can be N number of request at a time from NodeJS application. Below are the details and code samples:

1> Running cron Job every 5 mins:

const cron = require('node-cron');
const request = require('request');
const otherServices= require('./services/otherServices');
cron.schedule("0 */5 * * * *", function () {
  initiateScheduler();
});

2> Get the list of elements for which I want to initiate the request. Can receive N number of elements. I have called request function (getSingleElementUpdate()) in the forEach loop

var initiateScheduler = function () {
  //Database call to get elements list
  otherServices.moduleName()
    .then((arrayList) => {
      arrayList.forEach(function (singleElement, index) {
        getSingleElementUpdate(singleElement, 1);
      }, this);
    })
    .catch((err) => {
      console.log(err);
    })
}

3> Start initiating the request for singleElement. Please note I don't need any callback if I received a successful (200) response from the request. I just have to update my database entries on success.

var getSingleElementUpdate = function (singleElement, count) {
  var bodyReq = {
    "id": singleElement.elem_id
  }
  var options = {
    method: 'POST',
    url: 'http://example.url.com',
    body: bodyReq,
    dataType: 'json',
    json: true,
    crossDomain: true
  };
  request(options, function (error, response, body) {
    if (error) {
      if (count < 3) {
        count = count + 1;
        initiateScheduler(singleElement, count)
      }
    } else{
      //Request Success
      //In this: No callback required
      // Just need to update database entries on successful response
    }
  });
}

I have already checked this:

request-promise: But, I don't need any callback after a successful request. So, I didn't find any advantage of implementing this in my code. Let me know if you have any positive point to add this.

I need your help with the following things:

I have checked the performance when I received 10 elements in arrayList of step 2. Now, the problem is I don't have any clear vision about what will happen when I start receiving 100 and 1000 of elements in step 2. So, I need your help in determining whether I need to update my code for that scenario or not or is there anything I missed out which degrade the performance. Also, How many maximum requests I can make at a time. Any help from you is appreciable.

Thanks!

  • 1
    please don't cross-post: stackoverflow.com/questions/53512457/… "Cross-posting is frowned upon as it leads to fragmented answers splattered all over the network..." – gnat Nov 28 at 9:38
  • @gnat I am new here! Should I remove this question? Please suggest – Vaidya Nov 28 at 9:40
  • getSingleElementUpdate(single element, 1) will be executed sequentially, but callbacks won't since each request is asynchronous so basically they will be launched all at the same time. Regarding the main question. It will depend on the number of sockets available in the O.S. if you are faster opening sockets than O.S closing terminated ones, you could exceed the O.S capacity to allocate new connections. – Laiv Nov 28 at 15:06

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.