What kind of bottlenecks are possible when implementing a distributed workflow using a central point of control?

Question

I was going through https://docs.microsoft.com/en-us/azure/architecture/patterns/choreography, the document mentions in the drawbacks of having a central orchestrator that it can become a performance bottleneck. This is something I didn't get, because even if we move towards the choreography pattern as described, any workflow will still have to pass through each and every service involved. I don't see how it can be a bottleneck unless it was making a parallelizable workflow sequential.

I understand the benefits of this, like reduced coupling and the ease of bringing in new components into the flow, but could not get the performance bottleneck part. Can someone please explain what I'm missing here ?

Answer 1

With a small number of concurrent workflows you won’t notice the difference. But imagine that you scale up to the trillions of workflows per day:

• Let’s take a workflow involving 10 microservices, and take 100 000 workflows per seconds. That’s 10 x 100 000 = 1 000 000 service requests that your orchestrator will have to coordinate. That’s as many requests and answers, so 2 million messages.

• You can of course scale up by cloning the orchestrators. With 100 orchestrators, each orchestrator has only to manage 20 000 messages. But you’ll have the constraint that one workflow needs to be managed by a single orchestrator. And each or or orchestrator will spend most a lot of time waiting for answers.

• Now imagine the same workload but without central workflow orchestrator: a design centered around event processing would use a distributed queue, and instead of making requests to the microservices, the microservices would each subscribe to the relevant events: in the same amount of time, you’d have only 1 million of queued messages. But the queue management is lighter thant the orchestration. And it allows to decouple the processing more. Moreover enqueuing and dequeuing can be managed lock-free. The workflow is split down into workflow-steps and each can be handled independently. So each microservice gets its 100.000 messages but without channeling them through a central orchestrator. This gives much more flexibility for distributing the processing across the nodes, and thus scalability.