In a case where we have multiple source of data, we need to :

  • persist each source of data in its raw form.
  • process the data source eventually with each other to transform them in a presentable state (into a queryable database).

The first approach I could think of would be to store the data generated by the source using a stream that will dump the data into a file system. Then once this is finished for a part of the data I would read this new part with another computing engine able to transform and write the data as I wish. In this approach, identifying the new part of the data is blurry, I don't really know how to do that once the data is persisted to a file system. There are some tools that monitor the state of the file on a file system but it seems tricky.

The other approach would be to handle the raw persisting and the transformation in the same stream or in stream connected one after another. With this approach, I will at least always have one stream component that will need to 2 things and I don't like it much because I believe that one stream component should have the responsibility to do only one thing, in my case either transforming the data or persisting it raw.

Is there other solution beside those 2 approach ?

  • "With this approach, I will at least always have one stream component that will need to 2 things" - no, you only need a component which takes one stream as input and pipes it into two streams (duplicated) as output, just like the classic Unix tee command. If that's possible may depend on the framework you are using.
    – Doc Brown
    Jun 3, 2021 at 8:06
  • I will have 3 stream components. Stream 0 that will pipe data to stream 1 and 2. Stream 1 will dump to hdfs, stream 2 will execute the computing on the data, right ? Jun 3, 2021 at 8:18
  • 1
    That's the idea. Note I am not an expert on big data stream processing, but cloning a stream into two is a typical building block for pipelines. You need to check if your framework provides such a tool.
    – Doc Brown
    Jun 3, 2021 at 8:27
  • Aany idea is welcome, I'm using flume, the issue I have with that is that I'm using a way of sinking the data using multiple channel and sink, but for the same stream of data. It works to write it to hdfs this way because I point to a folder but if I want to pipe those multiple sink to another flume I can't because it the source of this other flume can connect to only one sink. Jun 3, 2021 at 8:39
  • Then maybe you should ask a specific question on Stackoverflow about flume (or, first, check if someone has already asked about this topic there in the past)?
    – Doc Brown
    Jun 3, 2021 at 8:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.