0

please note that while I refer to a specific web application framework in the following, the problem also arises with most other web application frameworks I know so please don't be afraid to reply even if you don't have experience with NestJS.

I have a web application with NestJS that runs on Azure Appservices (or AWS ECS). There are at least 2 instances active at the same time.

The application has a public RESTful endpoint allowing people to POST orders via HTTPS. These are then stored in a database table. The new orders shall be exported to a CSV file and the CSV file once a day at a fixed time to to orders processing department. The CSV must only be emailed at most once per day.

Currently this does not work because the export is initiated by a @Cron (from @nestjs/schedule library) decorated method in a service class within the application. As there are multiple instances of the docker container, the export is initiated once per running instance instead of only once per day.

My initial idea was to create another REST endpoint that is called once per day by a AWS lambda or Azure Functions function with a cron trigger for the lambda function. This endpoint will then invoke the export functionality.

As the web application instances run behind the load balancer, the HTTP POST request to start the export procedure would only be forwarded to one instance.

The only downside I see is that the lambda could be a single point of failure. If the cron trigger for some reason does activate or the HTTP request is lost somehow, the export would not be done right away. But if the HTTP request to trigger the export would fail or not be sent at all on one day, it could easily be resent by manually invoking the lambda function during the day. Another "downside" of this A very small additional cost for the lamdba function resource (0,000x cents?! if already above free tier limit)

What the other developers suggested was to use the @nestjs/bull module which makes use of redis to synchronize a queue across all connected instances.

This has multiple major disadvantages in my opinion:

  • A redis cache needs to be served with a good enough availability -> bigger additional infrastructure cost per month
  • More code needed for the wrapping business logic that sets up the queues at app startup (@OnModuleInit in NestJS) compared to a small lambda function (~10 LoC) combinend with an additional HTTP POST endpoint (at most 15 LoC in NestJS)
  • NestJS specific problems: I need THREE additional dependencies: @nestjs/bull, bull, ioredis (the latter to import Redis.RedisOptions interface in the app.module)
  • Default BullModule implementation lets app startup fail when redis is not available -> additional single point of failure (could potentially be circumvented by a custom wrapper module but this requires more code again)
  • Migration to other web application framework would be harder because the new framework would need to provide an implementation to consume the jobs from the redis queue. Quite unlikely that there would be an existing component that can also process the cron jobs stored in @nestjs/bull's data structures within redis. An HTTP POST API on the other hand can be replicated very easily using most web application frameworks (Spring, NestJS, Django, Golang Frameworks, etc.)

Downsides both solutions have:

  • Additional terraform resource (lambda function or redis cache)
  • Increased overall system complexity

Question 1:) What am I overlooking?

Question 2:) What other advantages / disadvantages do you see in both approaches?

Question 3:) what other approaches are there with having even fewer disadvantages?

3
  • This is getting far from the existing architecture, but I'd argue letting the orders people pull their own CSV (maybe even with some custom filters!) rather than trying to push it on a timer is something to consider. I don't know the needs and I've done both before, but for "business process" reasons I've found the pull-on-demand to be less failure-prone
    – user356474
    Feb 29 at 0:37
  • This would require us to offer the orders processing department a web UI or something where they can start the export and then download the file or so. This is out-of-budget ;)
    – dfsg76
    Feb 29 at 7:27
  • 1
    There are so many ways around this: Leader election, Database based locking/tracking, Distributed cache/queue, in essence a separate, singular point of reference for service nodes to resolve conflicts on.
    – S.D.
    Feb 29 at 8:06

1 Answer 1

5

What ... disadvantages do you see in both approaches?

You're thinking too hard. This is a solved problem.

what other approaches are there with having even fewer disadvantages?

Do ACID. Your problem statement makes it clear that you wish to have ACID guarantees. You might use e.g. Amazon RDS for Postgres, a clustered database offering.

Create a table with PK of (day, status), mapping to task_guid and timestamp columns. Daily 1am cron job should perform these steps in order:

  1. Roll a new task GUID to identify itself.
  2. Issue a SELECT query to identify the first date that is (A.) in the past and (B.) lacks a successfully sent email. Typically this will be "yesterday", but after disaster and a week of downtime it could correspond to a few days ago.
  3. Issue a SELECT query to report on recently posted orders.
  4. Do some trivial healthcheck API call to verify your SMTP service is available. Or send yourself a "please ignore" test message.
  5. INSERT (day, "about to send") task_guid and timestamp, and COMMIT.
  6. Do an API call to sendgrid or whatever SMTP service you're using, to send the CSV report. If the service returns fatal "not sent" error status then DELETE today's row, send an alert, and abort right now.
  7. INSERT (day, "sent") task_guid and timestamp, and COMMIT.

Step 5 is where one task will lock out another task with a mutex -- it is the step that matters. Things could afterwards potentially go south, and there is little to be done about that if your SMTP service doesn't support idempotency. Step 7 is really just an audit log, so you can assess how things went.

It would be convenient to randomly jitter your timers (sleep) so nodes don't all attempt to do this simultaneously. If running the report takes a long time or is otherwise expensive, we could optimize by querying whether there's already a "sent" record for today, which could let us drop out at once. At least one node should schedule a 2am run, "just in case" things didn't go smoothly at 1am.

Suppose two tasks are racing. They execute steps 1 through 4, which is fine, creating no side effects that we care about. At step 5, one of them will win, and the other will suffer a "unique constraint violated" DB diagnostic. At which point he just logs the event and drops out, confident that the other guy won.

Let's say that sending an email typically happens in much less than ten seconds. Any cron job that finds an "about to send" records without corresponding "sent" record should sleep(20) then re-query. There are several cases:

  • An {"about to send", "sent"} pair appears in the table, so all is well.
  • We see only "about to send", so a sys admin should be alerted.
  • There's zero records for today, indicating that an error was received from the SMTP service, so this task should retry, perhaps after sleeping for a little while.

The weak link here is (non-idempotent) step 6, and there's little to be done about it, at least with most commercial SMTP services. After a successful COMMIT we intend to send an email, but we could fail at any instant, for example due to power fail.

Is this fundamental? No, not if the external service offers some limited amount of cooperation. For example the Streams API in Apache Kafka is able to offer some very nice end-to-end guarantees. The key ability is the external service should be able to de-dup based on seeing same GUID presented a second time. This is the sort of thing that, in principle, an SMTP service ought be able to do.


You don’t need an RDBMS. But your requirements make it pretty clear that, in the presence of potential reboots, you need persistence combined with reliable battle tested mutex. If you choose to provision those capabilities using another mature technology, good, more power to you.

6
  • 1
    Great answer! I even saw patterns where they would use the FOR UPDATE NOWAIT on a simple table "job" (id, name) and the instances would lock the row to do a kind of distributed locking so there is no race between them.
    – Matheus
    Feb 28 at 21:28
  • The focus here is too much on the database.
    – dfsg76
    Feb 29 at 7:12
  • The current database is mongodb and another database is not acceptable in this scenario. Downside of the locking via database approach: with increasing number of instances, the database load will increase unnecessarily because every instance will issue a write operation at the same time when the cron triggers.
    – dfsg76
    Feb 29 at 7:18
  • 1
    This is done at most once per day, from a pretty limited number of instances (not hundreds I suppose). There's absolutely no reason to be concerned about performance or database load. Feb 29 at 10:14
  • 1
    @dfsg76 Databases are so much the solution that it's worth spinning up another database just to coordinate this. Deliberate jitter solves the "thundering herd" problem.
    – pjc50
    Feb 29 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.