Is node.js a good fit for background processing?

Question

I'm slowly learning node.js and have a small project I want to start. The project will have a lot of background processes (downloading data from external sites, parsing CSV files, etc.).

A big "win" for me and node is the fact it uses JavaScript for both client and server. I code in Java and JavaScript in my day job but am also pretty good at Ruby.

But, like I said, it seems attractive to use one language everywhere and JS seems to fit that bill.

However, I haven't had much experience in using JS for running background jobs. Ruby seems to excel at this. And I'm not opposed to using it. So what are your thoughts on going 100% JS for this? I realize very large projects require custom solutions. I'm just wondering if it's worth the effort. Or, should I just stick with Ruby on those kind of chores?

Opinions appreciated.

Thanks

Answer 1

It's particularly strong at handling a ton of file I/O and I would expect it to handle a ton of network communication well too. It seems particularly popular for socket-driven apps. The important thing to keep in mind is that if your needs aren't met by existing libraries (there are many) you may need to dive into some C which can be bound to JS commands. You can also spawn additional Node processes but I suspect doing a lot of that could get taxing (I'm assuming - might be wrong - there's a V8 instance spawned for each one of those).

JS is single-threaded and blocking, meaning nothing else can execute until a function call has completed. This was a desired feature of JS, essentially taking all the threading and queuing concerns out of your hands. The JS doesn't stop the C/C++ stuff from from running in a more multi-threaded fashion under the hood so JS's role is really more architecture/messenger. If you're image-processing, you're not going to want to handle that with synchronous JavaScript commands because everything else on your app or server will be blocked until it's done. The idea is that you call for an image to be processed by bound C/C++ functionality, and then respond to the 'done' event when the image is finished being processed.

This requires that the JS in any Node.js app be heavily event and callback driven or it will likely perform very poorly. So you won't see a lot of method calls in Node that don't get handed a function for later use. One thing that becomes very clear very fast in Node is that you're in for a world of ugly if you don't find a way to handle the callback pyramid. e.g.

//event CBs are more DOM-style than Node style and this isn't built-in Node file I/O
//keeping it simple and quick since I'll just get Node stuff wrong from memory
file.get('someFile.txt', function(e){
    e.fileObj.find('some snippet', function(e){
        someFinalCallBackHandler( e.snippetLocations );
    } );
} );

Fortunately there are plenty of tools and examples out there for handling this better. Most tend to revolve around promise mechanisms and simply chaining a series of functions meant to respond to each other's callback states in an array that does the ugly pyramid stuff for you under the hood.

Personally, I freaking love that we get JS at the high level and C/C++ closer to the chrome. It's the ultimate combo and it inspired me to start learning C. And don't let the lack of library potential freak you out until you've done some research. Node libraries are being produced at a very rapid pace and are maturing very rapidly. If you're not doing anything highly unusual odds are good somebody has it covered.

The biggest difference from Rails, is that JS is never likely to be on rails as it were. We tend to code for being able to have it however you want it very rapidly so there is the rope to hang yourself with factor and architecture has been pretty DIY in JS until more recent years. I call that freedom, but I realize that's not seen as ideal to a lot of devs.

Also, you will never ever have a "gem" problem in Node.js because you tried to install on something other than a Mac. Client-side web devs despise dependency issues and that's where a lot of Node's core is coming from. If it doesn't work out of the box in 5 minutes or less on every popular platform, we generally crumple it up and toss it. I have yet to run into a popular module that required I do anything special to get it working. The package system, is excellent.

But to answer your core question more explicitly/succinctly: Is it good with background processes?

Yes, Node basically IS background processes with a means of driving an app via events and callbacks.

Answer 2

One issue to be aware of is what occurs when processing large files in an asynchronous environment: if your input stream (a file) is faster than your output stream (the db) then you won't be able to handle the input data events quick enough. This will either overwhelm some piece of your system (output stream or memory) or cause you to lose data. For this reason, data processing asynchronously can be a little tricky. But as the article I linked to explains, the ability to pause the input stream makes it possible to throttle in a manner that fits your situation.

Answer 3

Node.js excels at IO. You are very unlikely to discover one day that your process jammed up since most of your threads are blocking in SQL calls.

However node.js is really bad at compute-bound work. When I hear "lots of IO" I think "yeah! go node!", but when I hear "parsing," I hesitate a little. I'm not sure if this is for any reason besides people not multithreading node properly, but thus far all of my product's compute-bound work happen outside of node.

Multithreading in node.js is tricky to set up right. Everything is single threaded by default and most code is written under the assumption that it will only run under one thread. You will certainly need to use domains to prevent an error on one thread from bringing down your whole application.

Also note that node may be a bit weaker in some enterprise capabilities. For instance, its logging libraries do not compare to Java's. At present there is no good logging framework that even supports and MDC, which in practice means you get to do a lot of var logPrefix = userId + ": " a lot.

I've also never run a private npm repo, you may need one of these depending on whether your code is proprietary.

Answer 4

If your background processes can run sequentially it can be pretty good. At my last position, I had to write a number of pre-processors, exports and translation utilities for many data sources. Using NodeJS was a breeze here.

If you aren't doing a lot of compute bound processing, simple manipulation of short strings, and integer parsing isn't so bad, if you need to manipulate images, it is probably not the best tool (though there are callable wrappers and modules that can work well).

Advice, stick to modules that use streams. This can make it easier to pipe your processing to modules for that particular step. If you look at how event-stream is used in gulp-jade for the gulp build tool for example, you can see how capable it is.

For CSV, you can use node-csv, which is pretty good at establishing a base for piping records to a processor stream.

For large-ish XML, where you want to do a single record at a time, I would look at node-halfstreamxml which reads through your XML stream using a SAX processor, and raises events for each node. I would wrap that into a read/write-stream so you can raise your desired matches. Many xml-object parsers in node will attempt to read/parse the entire xml at once, and for say 100mb of xml that gets huge... where the halfstreamxml will read as a stream.

NOTE: there are other processors like xml-stream which will use expat (C library) underneath, that can give more performance, but less portable without a build environment.

In general, it's been a real joy to use...