0

I have been working on a lambda that is triggered by a SQS with a batch (let's say of 5) and has automatic retry enabled.

The most common problem with this kind of setup that I have seen is that, if we have 5 events, and event number 1 and 2 are completed successful but event 3 fails then during the retry the 5 events are triggered again. When the lambda has side effect (sending email/writing to DB/etc) you don't want to trigger the successful events again as you don't want to have duplicated values.

The most common answer I have seen is to have a DB where you check before executing the event. Something like:

export const mySqsHandler: SQSHandler = async (event) => {
  for (const record of event.Records) {
    if (!db.has(record)) {
      await doStuffThatCanFail(record);
    }
  }
};

But this sounds over complicated.

Now, after looking for a while, I have seen this answer and I thought of coding the following example:

export const mySqsHandler: SQSHandler = async (event) => {
  const entries : DeleteMessageBatchRequestEntry[] = [];
  try {
    for (const record of event.Records) {
      await doStuffThatCanFail(record);
      // Add the successful message to a list
      entries.push({ Id:record.messageId, ReceiptHandle:record.receiptHandle });
    }
  } catch (error) {
    console.error('Failed', error);
    // delete all the message that succeed
    sqs.deleteMessageBatch({ QueueUrl: "url'', Entries:entries })
    // throw error to force the lambda to fail
    throw error;
  }
};

This example seems extremely easy and effective, so my question is, why haven't I seen it before? I have looked into SQS a lot and only one person suggested this.

Is there any big problem I'm missing? Will this even work?

1 Answer 1

0

The reason all the messages are put back in the queue when the lambda fails is the assumption is that there is a problem with the lambda itself. Say the network failed, or the machine crashed, or the lambda has a bug etc.

In these cases removing the messages from the queue would result in lost data. So unless a success is returned, via calling the delete method they are put back on the queue.

In this context:

  • Success means "I have done my job on these messages"
  • Failure means "I have not been able to process these messages, resend"

But that is not the type of error you seem to be talking about.

If you have some process that can fail your lambda needs to handle that failure while still returning a success.

You have two options.

  1. The failure is something that retrying can fix.

    You should fix whatever this is

    The lambda should internally retry, requeue should NOT be used as a retry method

  2. The failure is something that will always fail for this message

    There should be a defined process for dealing with this problem

    The Lambda should post the message to a new queue for that failure process

Either way the lambda completes successfully. No messages are requeued and all the messages are confirmed processed via the delete method.

Additionally, when dealing with queue systems, all your handlers should be written with the assumption that they may experience duplicate messages at some point. ie they should be idempotent.

This allows for error cases where a message is sent but the success message is not received, leading to the message being sent more than once.

4
  • 2
    "You shouldn't try and delete the messages from the queue manually." <- with SQS, you should - deleting a message (or more precisely, the specific ReceiptHandle you got) is the way to indicate that you have processed the message; see e.g. Sending and receiving messages in Amazon SQS Oct 18, 2021 at 10:22
  • oops my bad....
    – Ewan
    Oct 18, 2021 at 13:12
  • 1
    It is the law that everyone who has ever written an SQS client gets this wrong the first time and must spend an hour swearing at AWS "why do I always get the same .... messages every time?" until they read the documentation carefully. Oct 18, 2021 at 13:15
  • thing is I have done it before and dont remember this bit. I must have made a library and baked it in
    – Ewan
    Oct 18, 2021 at 17:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.