2

I am trying to decide on which approach to follow for long running tasks.

Web Client (user) story

  1. Web client connects to the server (REST API + WebSocket for real-time notifications)
  2. Web client POSTs a new resource intensive job request
  3. Server publishes the job on a queue
  4. Task worker consumes the job and processes it.
  5. On Task worker job completion:
    Option A. Task worker updates DB. DB trigger fires a function to send a notification to the web client through the WebSocket connection.
    Option B. Task worker POSTs job results to the Server. Server updates DB and sends a notification to the web client through the WebSocket connection.

I have implemented both approached in separate projects and I found both to be somewhat "problematic".

Downsides of each approach

Option A:

  • Task worker requires access to another service's DB, as a result coupling both services tightly to any change.
  • I didn't really like using post save DB trigger to fire custom logic (e.g. send notification)

Option B

  • The Server requires to expose a new endpoint, to which only the Task worker has to be able to send requests. (security concern)
  • If the server is down for any reason the job result is lost (since the task worker will try to POST and no one will be there to "hear")

The question

Is there another option I could make use of to enhance the current solution.

Architecture Alternatives

P.S. I am certain these are pretty standard/mainstream requirements for many webapp (real-time capabilities + REST + long running server side tasks), so there has to be some standard approach to tackle this.

I've looking into similar questions, such as this [1], this [2] and this [3], but I hope to get a more focused answer.

1 Answer 1

2

You already have one queue, just add another "job finished" queue and have your backend subscribe to it and forward the messages on the websocket connection.

enter image description here

Fixed professional data flow diagram

5
  • True. But this means my server will do more than one things. 1. Serve HTTP requests (REST + WebSocket connections) and 2. spawn a thread(is that what you suggest???) to periodically consume new messages (if any). IMHO its better for each service to do one thing (just like functions).
    – Gr3at
    Nov 27, 2022 at 14:58
  • if your server is holding the websockets connection its already got a thread running to manage that. its no extra complexity to have it subscribe to a queue for update events
    – Ewan
    Nov 27, 2022 at 15:00
  • although yes you could think about moving the websocket/push notification stuff to a different service. I don't think its super important though
    – Ewan
    Nov 27, 2022 at 15:01
  • That's what I am thinking of (i.e. moving the WebSocket to a separate service).
    – Gr3at
    Nov 27, 2022 at 15:03
  • similarly with the task consumer updating the db directly vs through a rest call. If its strongly coupled to the microservice i don't think its a huge issue for it to write direct, although technically it would be better to send the update to the REST service for it to write it.
    – Ewan
    Nov 27, 2022 at 15:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.