I have been working on a lambda that is triggered by a SQS with a batch (let's say of 5) and has automatic retry enabled.
The most common problem with this kind of setup that I have seen is that, if we have 5 events, and event number 1 and 2 are completed successful but event 3 fails then during the retry the 5 events are triggered again. When the lambda has side effect (sending email/writing to DB/etc) you don't want to trigger the successful events again as you don't want to have duplicated values.
The most common answer I have seen is to have a DB where you check before executing the event. Something like:
export const mySqsHandler: SQSHandler = async (event) => {
for (const record of event.Records) {
if (!db.has(record)) {
await doStuffThatCanFail(record);
}
}
};
But this sounds over complicated.
Now, after looking for a while, I have seen this answer and I thought of coding the following example:
export const mySqsHandler: SQSHandler = async (event) => {
const entries : DeleteMessageBatchRequestEntry[] = [];
try {
for (const record of event.Records) {
await doStuffThatCanFail(record);
// Add the successful message to a list
entries.push({ Id:record.messageId, ReceiptHandle:record.receiptHandle });
}
} catch (error) {
console.error('Failed', error);
// delete all the message that succeed
sqs.deleteMessageBatch({ QueueUrl: "url'', Entries:entries })
// throw error to force the lambda to fail
throw error;
}
};
This example seems extremely easy and effective, so my question is, why haven't I seen it before? I have looked into SQS a lot and only one person suggested this.
Is there any big problem I'm missing? Will this even work?