1

I built an audio processing web app using Rails. The user uploads a song to the website. The song is then decomposed into individual elements and then modified and recombined.

I am using a an open source command line tool that is being called from the rails controller.

My Problem: It takes around 2 to 3 minutes to do the processing and it consumes a lot of memory. The browser is in loading stage for 2 to 3 mins. And, this is just for 1 request from 1 user. I am using Amazon ec2 instance t1.large it is just not enough.

I am planning to use background processing but I don't want multiple requests to work the same time.

I want the first request to take 3 mins. 2nd request to take 3 + 3 mins. 3rd request to take 3 + 3 + 3 mins. So the website doesn't go down.

Also, the audio file is around 40 mb. Is it a good idea to use amazon s3 bucket? Or just increase the harddisk space and store audio files on my server?

The reason I don't want to use the amazon s3 bucket is because I don't want to transfer each file again from my ec2 instance into the s3 bucket, because it will add to the latency.

  • This might sound bad, but two make seperate servers. One is the Ruby on rails web server. It ensures that the file is uploaded, and makes a record in the database and adds a message to a queue. The second server is a task manager written in whatever language. It recieves a message from the queue, grabs the job record and performs the work. It can do this in anyway it wants to. When complete it marks the record as complete. If you want the user to be notified it will also add a message back to the web server which responds to the client. – Kain0_0 Jan 14 at 21:40
  • What kind of processing are you doing that takes two minutes for a 40 MB file? If performance (or lack of performance) costs you money, you could consider hiring someone to make that code faster. – gnasher729 Jan 15 at 17:15
3

Rails has a built-in framework for background task processing, so lengthy tasks don't block the controller from returning a response to the client. It's called Active Job. (See: https://guides.rubyonrails.org/active_job_basics.html) Using Active Job would be a better option than leaving your users' browser loading for several minutes.

You may still need some way to get an update to the browser when the background processing is done. There are lots of ways to do that, but the simplest would probably be:

  1. Insert a record in your DB for each job which is submitted
  2. When the background processing finishes, set a "done" flag (boolean value) on the job record
  3. Add an controller action which checks whether a job is done, and if so, returns whatever data needs to be displayed to the user
  4. Add some client-side JS which polls the server every N seconds until the job is done

It's not possible to say whether it is "better" for you to keep audio files locally or in S3 without more information. Probably you are not keeping copies of these files forever, but just while they are being processed? If so, what is the largest number which might need to be kept at one time? Does that fit in the available disk space, with plenty of room to spare? (It's a bad idea to run the free disk space on your server close to zero.) If not, how does the cost of a bigger EC2 instance compare to the cost of using S3?

You said that processing each file takes 2-3 minutes. I wonder if the time required to transfer the files to and from S3 would really add to that noticeably. Anyways, if you try both ways, you will find out for sure whether it makes a difference or not.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.