6

I'm relatively new to AWS. I'm working with what I think is a common pattern:

  1. Put file in S3 bucket
  2. Do something with said file in Lambda function

I see two options for making this link (ignoring SNS):

  • invoke the lambda when an S3 event occurs
  • send the S3 event to an SQS queue, which in turn triggers the lambda

It won't be handling a huge number of events to begin with, but the hope is to hook up a lot more buckets to this lambda in the future. Immediate invocation, message ordering, and speed/time is not of critical importance. However retries, capturing files that "fail", DLQs, and all that good stuff is important.

I'm leaning towards the SQS route. I think it fits better with my requirements, it's the one I've managed to make a working terraform module for, and I don't think it will add to my bill in any significant way.

Is this a matter of opinion or is there an objectively better option here?

  • why do you ignore SNS? – Ewan Jan 25 at 8:50
  • I don't see how s3 -> sns -> lambda is functionally different from s3 > lambda. – moarCoffee Jan 25 at 11:37
  • it might well be identical under the hood. Just curious as to why you explicitly rule it out. I would use it by default as it acts as a abstract layer between s3 and whatever you want to attach to the event – Ewan Jan 25 at 13:09
  • This is a fair point. I was perhaps too quick to dismiss the option. – moarCoffee Jan 27 at 15:43
5

I think the most robust approach is

S3 -> SNS -> SQS -> Lambda

SNS gives you that pub sub endpoint so you can attach more things to the event if required.

With SNS you can publish a message once, and deliver it one or more times. An SNS topic is an access point that recipients can dynamically subscribe to in order to receive event notifications.

SQS gives you the error handling and guaranteed delivery

If it is critical that all published messages be successfully processed, developers should have notifications delivered to an SQS queue (in addition to notifications over other transports).

However, there are reliability options with all combinations of choices. You would really need to have very specific requirements in order to say one way is the right way for you.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.