-2

I have a software application where we have 2 services, say A and B.
A delegates some long running work(tracked by an id) to B which is done asynchronously. They both talk via HTTP.
Post submitting the job, A periodically does getStatus calls passing the id.

Given that any of A or B can die, I need to handle failure scenarios.
I was thinking of using above getStatus calls as 2 way heartbeats which would behave as following for unhappy cases(and their handling after ->):

1. A unable to reach B -> heartbeat would timeout/connection refused, leading A to think work has failed
2. A is able to reach B but B has crashed and restarted -> heartbeat is treated as unknown, leading A to think work has failed
3. A dies/becomes unresponsive -> B misses few heartbeats and cancels its work(via timer), another A would reschedule this

Does above heartbeat handling looks like a good approach for A as well as B to know about each other's liveness?

Improve this question
3
  • 2
    Typically, asynchronous work is routed through a queueing system. In this case you are guaranteed that B would get the message to start the work regardless as long as A successfully sent the message. This avoids the need to have heartbeats to track liveness. If A wants to know status of a long running job, then it can query B using the job ID.
    – Berin Loritsch
    Jun 22, 2022 at 20:05
  • You are right, its just that there is one synchonous step in the interaction and hence the design is such. The same could have been implemented using 2 queues. But my question is more about liveness tracking.
    – Saurav Prakash
    Jun 22, 2022 at 20:10
  • 1
    Please don't edit questions so as to invalidate existing answers, especially if you're going to remove so much detail that the question becomes so broad as to be unanswerable.
    – Philip Kendall
    Jul 11, 2022 at 16:44

1 Answer 1

Reset to default
1

Leveraging work I did with a custom UDP interface, there's a few properties regarding liveness checking you need to keep in mind:

  • Timeout means the other system is unreachable
  • Active refusal means it's there, but it cannot accept any calls
  • Successfully receiving a message means the other service is up and running
  • With UDP, you can send a message but there is no guarantee it was received
  • There is no guarantee the service will still be up after a successful heartbeat

With an HTTP protocol, you at least have the confidence that if you receive a status code, you have successfully connected with the opposite service. You will need to establish time thresholds where that liveness signal is still considered fresh. That really depends on your particular needs. For example, in my UDP based system I had the following thresholds:

  • less than or equal to 6s, the signal was fresh
  • between 6s and 12s signal was stale or indeterminate
  • over 12s the other system was considered offline

Those are likely too tight for your purposes. The tradeoff here is that the more time you give to what is considered a fresh liveness signal, the more uncertain that actually is... but the tighter you make that threshold the more at risk you are of giving yourself a Denial Of Service attack.

In your case I would have the following in both services:

  • Track the timestamp of the last time a successful HTTP message was sent
  • The same timestamp can be used for successfully receiving a message as well
  • Have a local function that uses the delta from the current timestamp and the last successful communication to indicate "Fresh", "Indeterminate", or "Stale" signal state
  • Have a timer trigger at 1/2 the "Fresh" threshold to invoke the liveness check... but every time you have an event that would update that local timestamp, reset that timer.
Improve this answer
2
  • Yeah i had the same in my mind where B would have a local timer to determine if A is gone. This timer would be reset after every heartbeat.
    – Saurav Prakash
    Jun 22, 2022 at 21:19
  • I would reset it every time there is a successful communication as well. That should count as a heartbeat.
    – Berin Loritsch
    Jun 23, 2022 at 11:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.