12

I've asked what is now a deleted-by-Community question on SO on why would someone use javascript's Promise.race, and a high rep user commented this:

If you have two services that compute some value you can query them both in parallel and use which ever value is returned first, rather than querying one, waiting for a failure, and then querying the second.

I've googled about redundancy and this use case in general but I couldn't find anything and, from my POV, it's never a good idea to just add workload to a server/service if you're not going to use the response.

Improve this question
5
  • Toy example: rather than always using quicksort, you copy the data, send it to a quicksort, and a mergesort, and a heapsort, ... etc. You don't have to inspect the input to see if it is a pathalogical case for any of those, because it won't be a pathalogical case for all of them
    – Caleth
    May 17, 2018 at 9:00
  • The Dean and Barroso paper The Tail at Scale calls a variation on this approach "Hedged requests". It also discusses the pros and cons of several related approaches to controlling long-tail variability in error rates and latency.
    – Daniel Pryden
    May 17, 2018 at 13:55
  • The second "server request" could be a fake. It could just want for 5 seconds and then return a placeholder response. That gets you a timeout on the real request.
    – user253751
    May 18, 2018 at 1:13
  • Order a Lyft and then order an Uber. Take whichever one comes first.
    – user2023861
    Jun 12, 2018 at 15:40
  • @user2023861 in this analogy, while one driver was pointlessly driving towards your location, he/she could have taken on another request instead
    – Adelin
    Jun 12, 2018 at 15:46

5 Answers 5

Reset to default
11

I would argue that this is more of an economics question. However, that is a judgement call that engineeres ought to be able to do. Hence, I am answering.

I am splitting my answer in four parts:

  • Risk management
  • Strategies
  • Costs
  • Intuition

Risk Management

So, sometimes your client fails to get a response from the server. I will assume this is not because of a programatic error (otherwise the solution is to fix it, so go do that). Instead, it must be because of a fortuitous situation beyond your control...

But not beyond your knowledge. You must know:

  • How often does it happen.
  • What impact does it have.

For example, if failing and retrying happens only about 2% of the time, it is probably not worth to address it. If it happens about 80% of the time, well... depends...

How much time does the client has to wait? And how does that translate into costs... you see, you have a small delay in a regular application, it probably isn't a big deal. If it is significant, and you have a real time application or an online video game, this will turn users away, and you are probably better off investing in more or better servers. Otherwise, you probably can put a "loading" or "waiting for server" message. Unless, the delay is really big (in the order of tens of seconds), then it can be too much even for the regular application.

Strategies

As I said above, there are more than one way to go at this problem. I will assume you already have the try-fail-retry loop implement. So, let us see...

  • Put a loading message. It is cheap, helps user retention.
  • Query in parallel. Can be faster, can still fail. Will require a redundant server (can be expensive), will waste server time and network traffic.
  • Query in parallel to stablish the faster server and use that from there on. Can be faster, can still fail. Will require redundant server (can be expensive), will not waste as much server time and network traffic.

Now, notice I say these can still fail. If we assume that a query to a server has a 80% chance of failure, then a parallel query to two server has a 64% chance of failure. Thus, you may still have to retry.

A bonus advantage of picking the faster server and keep using it, is that the faster server is also the less likely to fail due to network problems.

Which remind me, if you can figure out why the request fail, do so. It can help you manage the situation better, even if you can't prevent the failures. For example, do you need more transfer speed on the server side?

Some more:

  • Deploy multiple servers across the world, and pick server by geolocation.
  • Do load balancing on the server side (a dedicated machine will take all the requests, and reley them to your servers, you can have your parallelism there, or a better balance strategy).

And who said you have to do only one of these? You can put a loading message, query multiple servers that are spread across the wrold to pick the faster and only use that from there on, on failure retry on a loop, and have each one of those servers be a cluster of machines with load balancing. Why not? Well, costs...

Costs

There are four costs:

  • The cost of development (usually very cheap)
  • The cost of deployment (usually high)
  • The cost runtime (depends of the type of application and the business model)
  • The cost of failure (probably low, but not necesarily)

You have to balance them.

For instance, let us say that you earn about a dollar per satisfied user. That you have 3000 users per day. That the requests fail about 50% of the time. And that 2% of the users leave without paying when the request fail. This means that you are losing (3000 * 50% * 2%) 30 dollar per day. Now, let us say that developing the new feature will cost you 100 dollars and deploying the servers will cost you 800 dollars - and ignoring the runtime costs - this means that you would have a return of investment in ((100 + 800) / 30) 30 days. Now, you can check your budget and decide.

Do not consider these values representative of reality, I picked them for math convinience.

Addendums:

  • Remember that I am also ignoring the details. For example, you may have little deploy cost but be paying for CPU time and you need to consider that.
  • Some clients may appreciate if you do not waste their data package in redundant requests.
  • Improving your product may help to bring natural advertisement.
  • Do not forget opportunity costs. Should you be developing something else?

The thing is, that if you consider the problem in terms of balacing costs, you can make an estimate of the cost for the strategies you consider, and use this analysis to decide.

Intuition

Intuition if foster by experience. I am not suggesting to do this kind of analysis every time. Some people do, and that is ok. I am suggesting you to have some understanding of this, and develop an intuition for it.

Futhermore, in engineering, aside from the knowledge we get from actual science, we also learn in the practice and compile guidelines of what works and what doesn't. Therefore, it is often wise to see what the state of the art is... although, sometimes you need to see outside of your area.

In this case, I would look at online video games. They have load screens, they have multiple servers, they will pick a server based on latency, and they may even allow the user to switch servers. We know that works.

I would suggest to do that instead of wasting network traffic and server time on every request, also be aware that even with redundant server, failure can happen.

Improve this answer
1
  • 2
    I don't think I need to say it but this is a great answer :) I knew I'll accept it at the first 10 lines, but I gave you the opportunity to still fail and read it to the end. You didn't
    – Adelin
    May 17, 2018 at 8:40
9

This is acceptable, if the time of the client is more valuable than the time on the server.

If the client needs to be fast and accurate. You can justify querying several servers. And it is nice to cancel the request if a valid answer is received.

And of course it is always wise to consult the owners/managers of the servers.

Improve this answer
2
  • Why do you need to cancel the request? Surely that is subjective.
    – JᴀʏMᴇᴇ
    May 17, 2018 at 8:15
  • @JᴀʏMᴇᴇ, That is build in paranoia. I once worked with a system that did not clear its queue and it did crash when the queue was full (Yes it was professional software).
    – Toon Krijthe
    May 17, 2018 at 8:21
4

This technique can reduce latency. Server response time is not deterministic. At scale it is likely there will be at least one server showing poor response times. Anything using that server will, therefore, also have poor response times. By submitting to several servers one mitigates the risk of talking to a poorly-performing server.

The costs include additional network traffics, wasted server processing and application complexity (thought this can be hidden in a library). These costs can be reduced by cancelling unused requests, or waiting briefly before sending a second request.

Here's one paper, and another. I remember reading a Google paper their implementation, too.

Improve this answer
2

I mostly agree with the other answers, but I think this should be exceedingly rare in practice. I wanted to share a much more common and reasonable example for when you would use Promise.race(), something I happened to use it for a couple weeks ago (well, python's equivalent).

Say you have a long list of tasks, some which can be run in parallel, and some which must run before others. You can start all the tasks with no dependencies, then wait on that list with Promise.race(). As soon as the first task completes, you can start any tasks that depended on that first task, and Promise.race() again on the new list combined with unfinished tasks from the original list. Keep repeating until all the tasks are done.

Note Javascript's API isn't ideally designed for this. It's pretty much the bare minimum that works, and you have to add quite a bit of glue code. However, my point is that functions like race() are rarely used for redundancy. They are primarily there for when you actually want the results from all promises, but don't want to wait for all of them to complete before taking subsequent actions.

Improve this answer
4
  • The problem is that, at least with Javascript’s Promise.race, you actually start the task each time you execute the race method. It won’t be on the unfinished task, it would be a new set of tasks, with no regards of what was ran before (unless you implement that logic at the task level). The original list is forgotten otherwise, and only the return value of the first task remains
    – Adelin
    May 18, 2018 at 19:25
  • 1
    Promises in Javascript are started eagerly, when new Promise is called, and not restarted when Promise.race() is called. Some promise implementations are lazy, but eager is much more common. You can test by creating a promise in the console that logs to the console. You'll see it logs right away. Then pass that promise to Promise.race(). You'll see it doesn't log again.
    – Karl Bielefeldt
    May 18, 2018 at 19:32
  • Ah that is true. But afaik the return value of the rest of the promises except the first one is forgotten, with promise.race
    – Adelin
    May 18, 2018 at 19:34
  • That's why I said the API isn't ideally designed. You have to store off the original set of tasks in a variable somewhere.
    – Karl Bielefeldt
    May 18, 2018 at 19:34
1

In addition to the technical considerations, you may want to use this approach when it's part of your actual business model.

Variations on this approach are relatively common in Real-time bidding on ads. In this model a publisher (ad space provider) will ask advertisers (ad providers) to bid on an impression by a particular user. So for every such impression, you would query each of the advertisers subscribed, sending a query with the impression details to an endpoint provided by each advertiser(or alternatively, a script provided by the advertiser running as an endpoint on your own servers), racing all these requests up to a timeout (e.g. 100ms) and then taking the highest bid, ignoring the others.

A particular variation of this that helps reduce the client's waiting time is to have the publisher allow for a minimum goal value for the bid, such that the first advertiser bid that surpasses that value will be immediately accepted (or, if none of the bids surpass the value, the max one will be taken). So in this variation, the first arriving query could win and the other discarded, even if they're as good or even better.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.