An application App-a I maintain publishes some events to an event bus for other team's application App-z. So far, the events were published right after being generated.
Now, I need to delay events ~30 minutes before publishing them. This is fairly easy to implement, but my concern is, that when App-a shuts down (restart, new version, etc), which typically happens a couple of times a week, these events will be lost.
What are strategies to deal with this situation? I have a couple of ideas, none of which I like too much:
- Persist the events somewhere, then have a scheduled job check the event dates and publish the ones older than 30 minutes. Cons: so far the application App-a doesn't persist anything so I would need to create & manage a new storage, schema, driver... Plus I would also need a new application to run the scheduled job.
- Have an intermediate service App-k doing the delay. The original application App-a would publish them as currently, then App-k can consume the events, wait for 30 minutes, publish them for App-a, and only at this point acknowledge event consumption to the event bus, to avoid having the same problem (losing in-memory events on shutdown) in App-k. Our broker is not Kafka so acknowledging an event is not as simple as committing and offset, so I'm not even sure waiting 30 for the ack is possible.
Before I get into any complicated and potentially bad mechanism I would like to hear suggestions for this problem.