I'm starting a project that will have a web front end for the users coupled with a database. There will then be a stand-alone service running that will, on a specified interval, poll an API and update the database with the changes it found.

What I need advice on is how the service should be setup. When a user creates an account on the site, I"ll need to create a "hook" in the service that will then go poll the API for that specific user.

I looked at building the service using NodeJS because I could use the non-blocking structure of it along with callbacks then the API returns data that needs to be reflected in the database. I don't know how I will be able to create more monitors when a user creates an account since it's already running. Can NodeJS run in a continuous loop as as service?

I am familiar with .Net and could use a Windows Service, but how would that scale with polling when the user numbers grow?

  • Can you elaborate a bit more on the architecture. Is the service on the same client machine where the user created the account from? What does the service provide? – Marcel Feb 15 '13 at 19:00
  • I was thinking of having the service be totally separate (maybe even on a separate server than the website). This service would only poll an API for data for each user, then update the database with data it gets from the API (the API is in-house so rate limits aren't an issue). It would have to run constantly as it would have to hit the API at a specified interval and check the response each time for a specific flag. I guess I'm more asking what would be best suited to build this service so that it would be able to run constanly and each user that signs up would get it's own check. – Justin Chmura Feb 15 '13 at 20:17

I'm still a fuzzy on exactly what you are asking, but based on your comment, I think I can offer some advice. As I understand it, you're mainly asking what you can be called by a "hook" and then ran in the background (not in real-time) to perform some sort of db-related update based on calls made to an in-house API. I imagine a simple diagram like the following:

 +----------------+                    +------------------+
 |                |                    |                  |
 |   web app      |    after-signup    |    background    |
 |                |+------------------>|    service       |
 |                |   async-call or    |                  |
 +----------------+      "hook"        +------------------+
                                      +  ^       +
                                      |  |       |
                                      |  |       |
                                      |  |       | update db
                             call API |  |       |
                                      |  |       |
                                      |  |       |
                                      v  +       v
                               +----------+     +------------+
                               |          |     |            |
                               |  API     |     |   Database |
                               +----------+     +------------+

So, in regards to how this "async-call" is made, there are a couple of common options that come to mind.

1. Using a Queue

Perhaps something like Redis or RabbitMQ. In this scenario, after the user has signed up, you're web-application will push some values into the queue (all the data the background service would need). Your background service would then be a program that simply watches the queue in a continuous loop and processes an item when something is found on the queue.

While this is a very simple approach, it does require that you install some sort of queue server inside your infrastructure. In that case, the diagram would change sightly:

 +----------------+                 +---------+             +------------------+
 |                |                 |         |    pop      |                  |
 |   web server   |    push         |         |<-----------+|    background    |
 |                |+--------------->| queue   |+----------->|    service       |
 |                |   async-call    |         |   process   |                  |
 +----------------+                 |         |             +------------------+
                                    +---------+             + ^       +
                                                            | |       |
                                                            | |       |
                                                            | |       | update db
                                                   call API | |       |
                                                            | |       |
                                                            | |       |
                                                            v +       v
                                                     +---------+     +------------+
                                                     |         |     |            |
                                                     |  API    |     |   Database |
                                                     +---------+     +------------+

Fire and Forget

The other approach is to setup a simple web-server as your processing service. All your web-application will need to do is send a fire and forget HTTP request to your service. By fire and forget I mean that you do not wait for a response when you make the HTTP call. You'll have to check what libraries you'll need to use for your language of choice to make this happen.

Then your background service, another web service in this case, will accept the request and start it's processing. Since the calling service is not waiting for a response, there is no rush here.


In either of these scenarios, you can use NodeJS if you'd like. It will work fine for both approaches. However, Ruby's EventMachine and Python's Twisted frameworks will do equally as good if evented IO is your goal, in which case I would have to assume that you are planning on handling a high-volume of requests/queue-items from your we application.

Let me know if this answers your question. Like I had mentioned previously, the question seems kind of broad but I answered what I could.

  • Actually, this was exactly what I was looking for. Sorry I didn't do a good job of expressing what I was trying to do. The only question I have left is when I setup the user in the queue and the service then processes it, would I need to setup a thread for each user since each user will have it's own call to the API? I'm also going to investigate the API to see maybe I can do batch calls to somewhat get around that issue. But thanks for the in depth answer! – Justin Chmura Feb 19 '13 at 14:18
  • If it is a queue then processing happens async which means you can do one at a time if you'd like or you can spawn up many threads. However, since it is a queue, you do not need to worry about there being one thread per user-in the queue (they can wait). – John Feb 19 '13 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.