I need to compute the value of a row R on a table based on rows queried on another table B. I need to do it in a queue worker because the computation of the value can be slow. A message is queued everytime B change.

If the two workers start the computation of row R almost at the same time, and the data on table B changed the final value of row R may be wrong. I just want the final value to be the right, I don't care if I jump a computation.

To do that I can :

  • Do optimistic lock on row R. Before the computation I read the version of R and when I update I do where version = ?. It ensures there are an independent computations n times if table B change n times (not 100% sure about that ^^). And I re-queue if a transaction fails.
  • Store a lock in etcd or something like this, the advantage is to avoid useless computations... And I re-queue if the row is locked
  • Use a FIFO queue (but I would like to avoid that)
  • maybe something else?

What do you think is the best option?

  • It depends on the odds that values in B change during a computation, and how much you value stale data in R. At one extreme (low rate of change in B, stale value worthless) you'd choose a very different method than at the other (high rate of change in B, recent values of R could be used). Can you elaborate on this? – user44761 Mar 31 '18 at 16:59
  • @Tibo Each value on table B is associated with an user. Most of the time the value only change when this user does something. And the row R is based on a custom formula for this user. So most of the time the computation is done before the user do something else causing table B to change. But sometimes row in B are massively updated. So everything can change very quickly. The only thing I care about row R is that, at the end, the value is correct. It's better if R get a new value for most change in B but I don't care if it jumps values – user3803241 Apr 1 '18 at 7:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.