1

I want to import some data from various locations and pump it into MongoDb and some cloud services.

The data starts off as JSON, I read that JSON, create some objects in mongo, go to the next level in the JSON, create some objects in mongo with a reference to the parent level, then the next level, do the same then search some REST services, hydrate those bottom level objects in Mongo, pull some images from another service, push them to a cloud service add a reference to the bottom level objects in mongo... etc etc

This all needs to happen sequentially because of the structure of things.

I tried doing this in a monolith node program, going to the next step in the promise returned. But this seemed nasty and also was quite buggy. The code was fugly even to look at and was basically an embarrasment due to the pure hecticness of the nesting and total disregard for all design patters since the dawn of time.

Since then I've been reading up on the Streams API. But since I am a nubz to NodeJs I don't really know if my plans here are an abuse of Streams or in fact what they were made for?

My design would be:

For each step in the process - create a class that can take some data, work on it and then omit it once it is done.

from the initial pull of JSON I can pipe the results into a chain of these classes, each class can 'visit' the JSON chunk and work on it as it sees fit. By the end of the chain all of the JSON would have been consumed and all of my objects hydrated and processed.

Because streams are back pressure aware it will go at it's own pace feeding in data when it's ready for the next step to take care of things. Does that sound like cray cray or am i on the right track?

[UPDATE]

I have just seen this which is probably a far less complex route to take:

https://www.npmjs.com/package/async#waterfall

  • It's OK to answer your own questions. Just be sure to include a summary and not just a link. See the help topic 'how to answer' for more info. – Dan Pichelman Feb 13 '15 at 14:57
  • Didn't mean to answer my question -was just adding to the options for solutions I could take. Not that there is one right way - but I wanted to offer it to the mix :) – iwayneo Feb 13 '15 at 15:00
  • Fair enough. I probably skimmed the question too quickly. For others reading these comments, see Can I answer my own question?. – Dan Pichelman Feb 13 '15 at 15:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.