1

I am writing an application that will crawl web pages and analyze data, and I'm wondering if I can host this as a cloud-based service.

The main reason I'm looking at cloud hosting is that users may process thousands of pages at once, and I need to return data as quickly as possible. The user will have software installed on their computer that will communicate with this application.

If, for example, each page takes 10 seconds to process, is it possible to set up auto scaling so that a user can process 1, 10 or 10,000 pages in the same amount of time (ie: 10 seconds) by simply adding and removing instances as needed? Note: Each process requires little RAM/CPU resources, so it will be possible to process multiple pages in a single instance but for this example, I'll simplified it to 1 instance per process.

If so, which cloud-based service is best suited for this task?

Improve this question
1

1 Answer 1

Reset to default
3

The big difference you are going to find between the platforms is how the vendor abstracted the environment. It will determine how you are going to develop your application or services. You can think of that as the API (of sorts for the application). The servies that you use within each vendor's context is up to you.

AWS doesn't really abstract much at the platform level. You basically get a machine where you install your software. It is up to you to write all of the supporting code. The services include storage, etc.

Azure provides a framework for you to run your code as a background or web service. It takes over and scales out from that perspective. You do not need to create the supporting os software to run your application. The services include storage, etc.

I haven not personally used GAE (yes to AWS and Azure) but it look like Google provides another framework and does not allow AWS like machine level access. Their environment appears to be more like the one hosted by Azure. They do have an interesting URLFetch service but include standard storage services as well.

If so, which cloud-based service is best suited for this task?

That would depend on the level control you would like to have? AWS provides the most control but would require more work. Azure and GAE provide more infrastructure that comes with limitations such as development languages. Azure is .NET while GAE has Java, Python.

I would suggest thinking about designing the application first and then choosing an appropriate language for development. That should give you a direction. If you are going to mix and match toolsets then it appears that AWS would be the way to go.

Improve this answer
3
  • AWS actually gives you easier to use services that GAE and Azure as far as I've seen; the machine for web servers is less abstracted out but if you're using .net IIS is a very easy installation. The idea behind AWS EC2 is you can run the same code on your machines as you would on EC2, and still use the other services (e.g. storage) that Amazon provide.
    – configurator
    Aug 16, 2011 at 17:24
  • And if you're in the US you can use AWS DevPay to automate purchasing instances or other services so you can send your customers to a link where they buy your application and pay Amazon directly for the services, and you set the price (and get that minus the AWS charges minus some commission).
    – configurator
    Aug 16, 2011 at 17:25
  • 1
    GAE is essentially a black box. You just deploy and it does its thing. It autoscales as needed and even shuts itself down when not in use. It's make admin a breeze, but it also removes all manual controls and leaves you at their mercy.
    – jiggy
    Aug 16, 2011 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.