I'm building a website that lets a user pay for a service that automatically does some video encoding for them.

Encoding takes several minutes. A naive solution would run each encoding job immediately. this could lead to several instances running and cause all of them to finish very slowly. the processing would probably affect serving requests.

As you can tell I don't do much web stuff. I'm sure there's some tool/approach that I'm supposed to be using in this situation, but I don't really know what it is

  • 1
    Search for "work queues" or "job queues". Names like Gearman, Celery, Beanstalkd should appear in your search results. – Mael Jan 23 '18 at 14:03
  • 1
    It's not really relevant to the question as it is posed but why is 'single server' a constraint? There are so many other downsides to that. Have you considered redesigning your solution to not be limited to one machine? You seem to be missing a plan for what you will do if your website is successful. – JimmyJames Jan 23 '18 at 17:34

This is a very high-level question, and it might get closed as "too broad". But in short:

What you need is an asynchronous solution: a request doesn't start the expensive job, but simply plans to do it and returns a ticket under which you can check for its stats. If you stay with only one server, you'll need to use a queue of some sort to ensure that jobs are processed one after another and each ticket is closed when its encoding job is done. The user must then download the result in a separate request, presumably by specifying their ticket number and their credentials again. If you're generous you might also want to give the customer an estimate of how long it will be before their job is done.

You will probably get better answers and more relevant search results by naming some of the italicized terms.


Basically you can't do this with a webserver (very effectively)

You need a second computer running a cron job/windows service or similar program. This is the "worker process"

Your webserver, takes the incomming request and saves it somewhere. Ideally a Message Queue, but a database will do in a pinch.

The worker process, waits for entries to the queue or database table. When it sees one it loads the data, completes the task and puts the result back on the database/queue

Now your user can load another webpage which queries the status of the job and gives him the results when its ready.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.