I want to import some data from various locations and pump it into MongoDb and some cloud services.
The data starts off as JSON, I read that JSON, create some objects in mongo, go to the next level in the JSON, create some objects in mongo with a reference to the parent level, then the next level, do the same then search some REST services, hydrate those bottom level objects in Mongo, pull some images from another service, push them to a cloud service add a reference to the bottom level objects in mongo... etc etc
This all needs to happen sequentially because of the structure of things.
I tried doing this in a monolith node program, going to the next step in the promise returned. But this seemed nasty and also was quite buggy. The code was fugly even to look at and was basically an embarrasment due to the pure hecticness of the nesting and total disregard for all design patters since the dawn of time.
Since then I've been reading up on the Streams API. But since I am a nubz to NodeJs I don't really know if my plans here are an abuse of Streams or in fact what they were made for?
My design would be:
For each step in the process - create a class that can take some data, work on it and then omit it once it is done.
from the initial pull of JSON I can pipe the results into a chain of these classes, each class can 'visit' the JSON chunk and work on it as it sees fit. By the end of the chain all of the JSON would have been consumed and all of my objects hydrated and processed.
Because streams are back pressure aware it will go at it's own pace feeding in data when it's ready for the next step to take care of things. Does that sound like cray cray or am i on the right track?
I have just seen this which is probably a far less complex route to take: