In a case where we have multiple source of data, we need to :
- persist each source of data in its raw form.
- process the data source eventually with each other to transform them in a presentable state (into a queryable database).
The first approach I could think of would be to store the data generated by the source using a stream that will dump the data into a file system. Then once this is finished for a part of the data I would read this new part with another computing engine able to transform and write the data as I wish. In this approach, identifying the new part of the data is blurry, I don't really know how to do that once the data is persisted to a file system. There are some tools that monitor the state of the file on a file system but it seems tricky.
The other approach would be to handle the raw persisting and the transformation in the same stream or in stream connected one after another. With this approach, I will at least always have one stream component that will need to 2 things and I don't like it much because I believe that one stream component should have the responsibility to do only one thing, in my case either transforming the data or persisting it raw.
Is there other solution beside those 2 approach ?