0

I've been having the hardest time understanding the difference between AWS Kinesis and Logstash.

From the description of Logstash it says

Logstash is a light-weight, open-source, server-side data processing pipeline that allows you to collect data from a variety of sources, transform it on the fly, and send it to your desired destination.

and from the description of Kinesis it says

With Amazon Kinesis, you can ingest real-time data such as video, audio, application logs, website clickstreams, and IoT telemetry data for machine learning, analytics, and other applications. Amazon Kinesis enables you to process and analyze data as it arrives and respond instantly instead of having to wait until all your data is collected before the processing can begin.

It's not obvious to me what the difference between them is as they both seem to be able to ingest data from many sources at once and can, in turn, process it.

Could somebody clarify here?

0

To my understanding, AWS Logtash is a sort of ETL, capable of intaking unstructured data from different sources for the purpose mentioned: Extract, transform and load the transformed data somewhere else. It's a collector that makes some changes to the incoming data, probably it normalises different data structures into a canonical one that fits our needs for mining or monitoring. But the data collecting and data mining happen at different times.

On the other hand, Kinesis, unlike Logtash, doesn't collect and transform the incoming data into something different. It's a support to perform analysis in real-time. It generates an additional stream of data based on the analysis and the results. In other words, data is not sent to Kinesis to be transformed and collected, it's sent to be analyzed as soon as possible for us to make decisions as soon as possible too. Probably, Kinesis allows you to send signals to other services based on the results of the analysis. With Logtash, we would have to wait for it to load the transformed data and then perform calculations; with Kinesis these things happen as the streams go.

Logtash

(input) ----> (Logtash : ETL)  ----> (Store)  <--- (Mining & monitoring)

Kinesis

(input) ----> (Kinesis : Analysis)  ---> (output) 
                    |
                    |
                    +-> (Metrics) <--    (Monitoring)
                    +-> (Metdata) <--    (ML/BI)
                    +-> (Statistics) <-- (BI/BI)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.