Invoking lambda directly from S3 vs going via SQS

Question

I'm relatively new to AWS. I'm working with what I think is a common pattern:

1. Put file in S3 bucket
2. Do something with said file in Lambda function

I see two options for making this link (ignoring SNS):

• invoke the lambda when an S3 event occurs
• send the S3 event to an SQS queue, which in turn triggers the lambda

It won't be handling a huge number of events to begin with, but the hope is to hook up a lot more buckets to this lambda in the future. Immediate invocation, message ordering, and speed/time is not of critical importance. However retries, capturing files that "fail", DLQs, and all that good stuff is important.

I'm leaning towards the SQS route. I think it fits better with my requirements, it's the one I've managed to make a working terraform module for, and I don't think it will add to my bill in any significant way.

Is this a matter of opinion or is there an objectively better option here?

Answer 1

I think the most robust approach is

S3 -> SNS -> SQS -> Lambda

SNS gives you that pub sub endpoint so you can attach more things to the event if required.

With SNS you can publish a message once, and deliver it one or more times. An SNS topic is an access point that recipients can dynamically subscribe to in order to receive event notifications.

SQS gives you the error handling and guaranteed delivery

If it is critical that all published messages be successfully processed, developers should have notifications delivered to an SQS queue (in addition to notifications over other transports).

However, there are reliability options with all combinations of choices. You would really need to have very specific requirements in order to say one way is the right way for you.