I am trying to decide on which approach to follow for long running tasks.
Web Client (user) story
- Web client connects to the server (REST API + WebSocket for real-time notifications)
- Web client POSTs a new resource intensive job request
- Server publishes the job on a queue
- Task worker consumes the job and processes it.
- On Task worker job completion:
Option A. Task worker updates DB. DB trigger fires a function to send a notification to the web client through the WebSocket connection.
Option B. Task worker POSTs job results to the Server. Server updates DB and sends a notification to the web client through the WebSocket connection.
I have implemented both approached in separate projects and I found both to be somewhat "problematic".
Downsides of each approach
- Task worker requires access to another service's DB, as a result coupling both services tightly to any change.
- I didn't really like using post save DB trigger to fire custom logic (e.g. send notification)
- The Server requires to expose a new endpoint, to which only the Task worker has to be able to send requests. (security concern)
- If the server is down for any reason the job result is lost (since the task worker will try to POST and no one will be there to "hear")
Is there another option I could make use of to enhance the current solution.
P.S. I am certain these are pretty standard/mainstream requirements for many webapp (real-time capabilities + REST + long running server side tasks), so there has to be some standard approach to tackle this.