2

I was thinking about experimenting a bit with using mapreduce and such on a dataset we have to see if we can benefit from it in any way.

What route should I go if I want to be able to instantly, without any major modification, deploy my findings to some kind of grid/cloud-system to see if there is any extra benefit to have it scaled in the cloud.

Improve this question

1 Answer 1

Reset to default
1

"The Cloud" is just a bunch of machines that you don't own. Running on those machines is no different than running on the machines that you do own, except that you pay for the time and bandwidth.

If you're using a tool like Hadoop, adding machines is just a matter of configuration (you don't mention platform, but it's the main player in the non-Google map-reduce space, which is just one of the big-data techniques).

My recommendation is that you work through the tutorial, get a single-node implementation of your application running, then immediately move to a cloud service like Amazon EC2. You can start with two small instances for about a dollar a day, and increase the number of machines from there.

As a cautionary note: unless you're dealing with 100s of gigabytes of data (and preferably terabytes), I don't think you'll see much benefit of either horizontal scaling or "the cloud." There is a not-insignificant time to setup and teardown map-reduce operations.

Improve this answer
3
  • Why immediately? Didn't you just say that it doesn't matter?
    – Robert Harvey
    Jun 4, 2013 at 17:37
  • Um, a cloud is a cloud whether or not you own the machines in question. To me, the distinction comes when you add a layer on top of the network of machines that essentially behaves like an operating system for groups of computers. But everyone has their own definition.
    – btilly
    Jun 5, 2013 at 5:59
  • @RobertHarvey I agree with the "immediately" bit. When you have one machine, it is too easy to write to a file in one place, and expect that file to exist in another. Once you've got 2 machines, you notice that you've introduced those dependencies because things don't work. But if the software scales horizontally, there should be no real difference between 5 and 500 machines other than that the second is faster. (Note I said "if" and the if in question is always false to some extent...)
    – btilly
    Jun 5, 2013 at 6:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.