Tweeted twitter.com/#!/StackProgrammer/status/419191194997235712
4 added 47 characters in body
source | link
  1. User selects files and requests a zip file to be created.
  2. Front-end sends a request to SQS with the files to be zipped, and the destination (on S3) where the final zip file will be stored.
  3. The queue worker uses long polling to check for new SQS jobs, and when. When a job is found, zipsthe worker downloads the files from S3, zips them up, and returns the finished productzip file to the specified S3 location on S3.
  4. Now that the queue has finished processing, alert the client that their job is done and their file is ready for download.
  1. User selects files and requests a zip file to be created.
  2. Front-end sends a request to SQS with the files to be zipped, and the destination (on S3) where the final zip file will be stored.
  3. The queue worker uses long polling to check for new SQS jobs, and when found, zips the files, and returns the finished product to the specified S3 location.
  4. Now that the queue has finished processing, alert the client that their job is done and their file is ready for download.
  1. User selects files and requests a zip file to be created.
  2. Front-end sends a request to SQS with the files to be zipped, and the destination (on S3) where the final zip file will be stored.
  3. The queue worker uses long polling to check for new SQS jobs. When a job is found, the worker downloads the files from S3, zips them up, and returns the finished zip file to the specified location on S3.
  4. Now that the queue has finished processing, alert the client that their job is done and their file is ready for download.
3 added 113 characters in body
source | link

I'm aware of theoretical solutions, so please be specific. Pointing I am open to any suggestions, regardless if they use SQS or any of the technologies/methodologies I have listed. Pointing to any real-world code examples of this sort of thing would be a big bonus.

I'm aware of theoretical solutions, so please be specific. Pointing to any real-world code examples of this sort of thing would be a big bonus.

I'm aware of theoretical solutions, so please be specific. I am open to any suggestions, regardless if they use SQS or any of the technologies/methodologies I have listed. Pointing to any real-world code examples of this sort of thing would be a big bonus.

2 edited title; added 28 characters in body
source | link

Best way to notify the client in real time that their queue (e.g. SQS process) job has finished?

  1. Use pub/sub to create a socket for the job, and return the response when complete.

    Use pub/sub to create a socket for the job, and return the response when complete.

    • From everything I have read, it is not best practice (and sometimes not possible) to create a large number of open pub/sub sockets/channels. In addition, how does SQS notify the open socket that the job is complete? This article regarding Amazon SNS looked promising at first, but they seem to infer that it is best practice to use a single SNS topic for the entire queue, not an SNS topic per job. This seems like a great approach if we were simply wanting to send an email to the user, letting them know their job has finished. However, I want to be able to update the user interface inside the web browser.
  2. Continuous polling by the front-end server for the finished zip file on S3.

    Continuous polling by the front-end server for the finished zip file on S3.

    • If there are thousands of simultaneous jobs, this would create thousands of requests. This approach doesn't seem very scaleable, and could potentially cost money by extra S3 api calls.
  3. Leave an open long-running request

    Leave an open long-running request

    • Due to the fact that if there are many jobs in the queue, the time taken to deliver the response could cause the request to time out. In addition this creates a lot of strain/wasted resources on the front-end web server. Again, not very scaleable.

Best way to notify the client that their SQS process has finished?

  1. Use pub/sub to create a socket for the job, and return the response when complete.
    • From everything I have read, it is not best practice (and sometimes not possible) to create a large number of open pub/sub sockets/channels. In addition, how does SQS notify the open socket that the job is complete? This article regarding Amazon SNS looked promising at first, but they seem to infer that it is best practice to use a single SNS topic for the entire queue, not an SNS topic per job.
  2. Continuous polling by the front-end server for the finished zip file on S3.
    • If there are thousands of simultaneous jobs, this would create thousands of requests. This approach doesn't seem very scaleable, and could potentially cost money by extra S3 api calls.
  3. Leave an open long-running request
    • Due to the fact that if there are many jobs in the queue, the time taken to deliver the response could cause the request to time out. In addition this creates a lot of strain/wasted resources on the front-end web server.

Best way to notify the client in real time that their queue (e.g. SQS) job has finished?

  1. Use pub/sub to create a socket for the job, and return the response when complete.

    • From everything I have read, it is not best practice (and sometimes not possible) to create a large number of open pub/sub sockets/channels. In addition, how does SQS notify the open socket that the job is complete? This article regarding Amazon SNS looked promising at first, but they seem to infer that it is best practice to use a single SNS topic for the entire queue, not an SNS topic per job. This seems like a great approach if we were simply wanting to send an email to the user, letting them know their job has finished. However, I want to be able to update the user interface inside the web browser.
  2. Continuous polling by the front-end server for the finished zip file on S3.

    • If there are thousands of simultaneous jobs, this would create thousands of requests. This approach doesn't seem very scaleable, and could potentially cost money by extra S3 api calls.
  3. Leave an open long-running request

    • Due to the fact that if there are many jobs in the queue, the time taken to deliver the response could cause the request to time out. In addition this creates a lot of strain/wasted resources on the front-end web server. Again, not very scaleable.
1
source | link