Can map-reduce say "Hello World"?

Question

Gathering that map-reduce is being used to process huge amounts of data, I set out to understand it.

My queries were:

• What class of problems does it aim to solve?
• How does it help breaking down of complex problems?
• Can I write a sample app, (locally!) to get started?

After a lot of googling, blogs and wikipedia, I know who is using it, who is selling infrastructure for it. But, I am still clueless about the above queries.

The closest I got is this tutorial on Amazon: "Contextual Advertising using Apache Hive and Amazon Elastic MapReduce with High Performance Computing instances" http://aws.amazon.com/articles/2855?_encoding=UTF8&jiveRedirect=1

This tutorial assumes understanding of online advertising and map-reduce, hence, tough to follow.

Anybody have any clues to how a programmer can get started on map-reduce?

Answer 1

Hadoop isn't a program so much as a software ecosystem. It contains a framework for the distributed running of map-reduce jobs over huge sets of data, usually read in from the distributed filesystem called HDFS. The real win of Hadoop is that you take the computation to the data rather than moving the data to the computation. The goal of Hadoop is to reduce and distribute I/O load. Hadoop also takes care of some fault tolerance and ensuring your jobs always complete quickly, even if you have a bad node in your cluster.

MapReduce aims to solve problems where you can easily split up data. Take perhaps the entire Gutenberg library. You can easily break these down upon word boundaries, or even chapter boundaries if you're smart enough. The idea being, you rely on associative/commutative operators as much as possible so you can shard the work infinitely then reduce it later.

As for MapReduce, in the WordCount example, say we do a frequency analysis on words with a length greater than four. In our Mapper, it's handed a set of data. You split your paragraph(s) on word boundaries and then emit a key,value pair which will later be sent to a reducer. So if we got the word "hadoop", we'd emit ("hadoop", 1) as we'd seen it once. We do this emission for every word greater than four characters long. These are then sent to the reducers. Each reducer takes charge of a particular key, so all the instances of "hadoop" go to a specific reducer. Since we emitted a bunch of ("hadoop", 1) keys we can then collapse these down into something like ("hadoop", 17) and have a single key value pair.

More or less, a Mapper is a transformation/filtering function. It doesn't have to be, but that's the general use case.

Apache Hive/Pig are simply abstractions which use MapReduce jobs to run queries over large sets of data, especially log files. They're a great way to extract analytics from real-time logging data from a cluster of webservers.

You can run this locally. Check out the Cloudera CDH releases, it can get you started almost instantly. By default, the Map-Reduce jobs run on your machine in series. You can configure a pseudo-distributed cluster on your local machine to take advantage of multiple processors, but the LocalJobRunner is built to test your ability to break up data and get the right answer, not for legitimate job running.

Cloudera also just released a SCM Express package which can let you install a cluster of up to 50 machines in almost no time flat.

Answer 2

If you want to write a simple map reduce program that runs locally look at hadoop streaming.

http://hadoop.apache.org/common/docs/r0.15.2/streaming.html

You can simply pipe in unix commands and have a working word count program only using linux commands. I recommend toying around with this and trying to build more complex mapreduce jobs. The benefit of map reduce is mainly the framework. If you want to toy around and learn how to write map reduce jobs, you should just take a look at this tutorial:

http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/

and build your jobs around that.