Not the most well-versed with AWS tech, and want to learn a bit about tradeoffs between different telemetry approaches.

We have clients that send requests to our system, and we want to produce both BI + Customer-facing dashboards based on logs data.

Right now, we stream logs data via fluent-bit to S3, and then using a series of Glue crawlers / ETL Jobs + Athena (for SQL queries), we visualize the information in Quicksight. The data (json entries) are usually 1-250kb in size.

What would be a motivating factor to move away from this approach to Redshift? My goal here is to understand if we should start considering other approaches, and what factors may trigger those considerations. Any other amazon tech we should use?

Couple of things we have noticed so far - glue crawlers and athena queries are slowing down, and we don't have that much data (maybe O(1-10M) log entries). Not sure if that's just because we need better data compaction practices or something.


  • Questions asking for trade-offs are off-topic here. Having said that, Redshift is a columnar data store (good for retrieving a large volume of specific columns). Which approach you use depends primarily on the shape of your data and what insights you'd like to get from it. You can get poor performance from any system. I'd look at data compaction before evaluating a stack change.
    – Dan Wilson
    Jun 16 at 2:47


Browse other questions tagged or ask your own question.