Scaling without queues?

Question

Whenever I am asked how I would scale out an application I inevitably tend toward a queueing model: split the application responsibilities into individual services and add queues between those services. This means you can spin up more or less instances of a given service as required and they simply pull from the queue to get work.

Get enough of these services with different roles and almost inevitably there is the creation of orchestration layer - something that understands the necessary flow of a work item through the queues and manages that end to end.

What are some of the alternative approaches for scaling out applications that don't use queues and end up with an orchestration layer?

Update based on comments

As @tofro pointed out, I'm probably talking more about elasticity instead of scalability https://stackoverflow.com/questions/9587919/what-is-the-difference-between-scalability-and-elasticity.

Here is an example. Let say I have a service that does video encoding. A user uploads a file, selects one or more different encoding formats (quicktime, divx), the file is encoded into the formats and the user can download the resulting output files.

One way to make this elastic using queues would be to have different services, QuickTimeEncoder and DivXEncoder, with queues QTEQueue and DivXQueue and put jobs on queues as required. More instances of the encoders could be added over time as demand changes.

Answer 1

In the absence of queues, you must spin up a new agent immediately for each new task to be executed; the new agent holds the task instance instead of a queue.

The Erlang programming language is capable of doing this, because it has the capability of spinning up millions of lightweight agents.

Note that Erlang also has a queue module, so it still gives you that option.

Answer 2

The easiest way to scale an application in my experience is to make it stateless and add instances. This is basic horizontal scaling. I an approach that is a little more sophisticated is to decompose the service across a your resources and then use orchestration to compose these various pieces into the needed functionality. There is no need for queues to do this. HTTP works just fine.

Queuing used to play a more prominent role than it does in contemporary design. When memory was highly constrained, it was important to be able to have a place to store a mass of transactions in order to push them through the small pipe of memory. Now we have 64 bit memory addressing and it's cheap and common to have more memory than you need. Systems with 1TB of RAM are nothing special.

The other aspect of queuing is that it could be used to manage IO waits. That is, you would have threads doing little pieces of work and moving the transaction on to the next thing instead of making blocking calls and waiting. With non-blocking IO libraries and other faster approaches to concurrency distributed queuing has become more of a niche approach. I think it still has a place in asynchronous processing but as @tofro mentions in the comments, the need for scalability doesn't necessarily indicate distributed queuing. There are lots of complications and problems that come with queueing as well as a lot of baggage about 'guaranteed delivery' which doesn't guaratee much in reality. I would avoid it unless you have specific problems that it solves better than the other techniques at your disposal.