We’re rewarding the question askers & reputations are being recalculated! Read more.

Hot answers tagged

7

Hadoop was originally written in Java, because it was used to "fix" problems in Nutch, which also was written in Java. Nutch, in turn, was written in Java because it was a write once run anywhere solution. As for whether C++ or another language would have been a better choice, that's definitely up for debate. With modern architectures, I'd trust Java or ...


7

When someone is asking about the certification question my response is always the following: What will this certification give you besides personal satisfaction? In my knowledge there are 2 certifications that had real money associating with having them in IT world: CCIE and old Citrix Administrator certification. Both of which had both theoretical and ...


6

I'd recommend keeping the data in HDFS and converting it to the Parquet file format. Parquet uses a concise, columnar representation of nested data and will reduce the I/O required for many of your queries. Once your data is in the Parquet format, I'd use Impala to issue SQL queries against the data. Impala implements a highly efficient execution engine for ...


5

The architecture is good enough to handle many requests per second, as long as you test it and profile it and it proves to handle the load that it is required to handle. Let me quote Donald Knuth, Computer Programming as an Art, 1974: The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at ...


5

Look at these key-value stores: Berkeley DB Java Edition, or JDBM (JDBM3 is the latest), or MapDB (JDBM successor). Tokyo Cabinet is not native Java but has a Java wrapper. For an overview see http://en.wikipedia.org/wiki/Dbm.


4

MapReduce actually has a grouping phase. The map phase essentially consists in transforming inputs into pairs of (key,value) elements. Because the reduce phase consists in "aggregating" all the values associated to the same key, you cannot avoid the need to group all values by key before the reduce phase. This may need a lot of time since values must be ...


3

I would recommend starting with Solr, then do your machine learning with Mahout and Hadoop. Solr will give you basic text analysis through word stemming, normalization (lower-casing), and tokenization. If you enable term vectors in the schema you can feed those directly into Mahout and experiment with the different algorithms there. A lot (maybe most) of ...


3

As with many complex technologies, Hadoop can be a challenge to learn on one's own. That is one reason that Cloudera's training courses are so popular. The courses for Developers, Administrators and Analysts are all hands-on courses taught by expert instructors. The certification provides the ability to prove that one fully grasped the concepts. It is ...


2

" I use JMS to send a message to Hadoop to handle a particular data, and then wait for it to come back to me" -- this is using an asynchronous protocol to implement a synchronous interaction. Is there any need for your program to wait for the reply? If there is couldn't you have another process handle the replies? There are several asynchronous patterns ...


2

My understanding is that, for a company investing in hadoop, training the workforce at their expenses is a must. They won't even be looking for already-certified workers. There just ain't enough around. "People who overheard the word hadoop at a conference" will be their typical interview subject. A certified professional with some experience instead would ...


2

Why does the design of HDFS have a single name node? Simplicity. According to http://hadoop.apache.org/common/docs/r0.20.2/hdfs_design.html#NameNode+and+DataNodes: The existence of a single NameNode in a cluster greatly simplifies the architecture of the system. The NameNode is the arbitrator and repository for all HDFS metadata. You can have a ...


2

But what is MapReduce itself? I think it's best to start from the basics: map is an operation that applies a single-argument function to a sequence of inputs, producing an equal-sized sequence of outputs. For example, assume an sequence of inputs [1, 2, 3] and a function f(x) => x * 2. The output would be [2, 4, 6]. As long as map is given a pure ...


1

MapReduce allows you to create a program that will run on a distributed data center. The program is divided in two phases. The first is sent to the servers where the data is and make a pre-calculation. The results of this phase is sent to the second part of the program to be consolidated. This works best with data processing where the first part results ...


1

The streaming solution will have lower latency, but I don't think that should be the primary consideration. Instead, I would look at overall system complexity. Whether streaming or batch the code will be pretty much the same, but with streaming you get better error recovery and don't need to setup a scheduler.


1

SRP happens at the Module (or library), Package (or namespace), Class, and Function level. What this means is: A Library has one reason to change. For instance, you build a network library. Valid reasons for it to change include: supporting a new protocol, fixing a bug in an existing protocol. Invalid reasons to change: client or server endpoints change. A ...


1

There are multiple things to consider in your case. What program are you using to create your dashboard? There are tools, such as Tableau that could help via creation of extract - though this may cost money. You could try using PostgreSQL - it is known to be faster than MySQL and it is free. You could set up your batch process to real time where it cleans ...


1

If you read the article you linked, it says Running a simple unit test on your desktop machine should highlight that creating 1x10^6 new String objects with random byte content is slower than using a single Text object and calling the set method to configure the underlying byte contents Well, that is self-evident. Creating a million new strings is ...


1

It sounds like the biggest problem you have left is to actually develop the categories and flesh them out. Given a set of categories, and a set of 'marker' words for each category. (You'll want to read about stemming (turn vegetables -> vegetable) and stopping (skip common, meaningless words the, etc). A good implementation of the stemmer you can find will ...


1

How to merge/sort/page through a huge amount of data? Well, for sorting, look at Quicksort if the data is more or less randomized, or Timsort if it's highly ordered. (Quicksort degenerates easily into horrible performance on highly ordered data.) For merging, there's a pretty simple algorithm for this: list comparison. Take two lists, list A and list B. ...


1

Update: There is an option of High Availability now. The HDFS High Availability feature addresses the SPOF problem by providing the option of running two redundant NameNodes in the same cluster in an Active/Passive configuration with a hot standby. This allows a fast failover to a new NameNode in the case that a machine crashes, or a ...


1

You might want to consider using something like the Mule ESB: http://www.mulesoft.com/mule-esb-open-source-esb It's open source, pretty lightweight, can host web services and supports full asynchronous processing of incoming events. You could start with just creating a lightwieght Mule web service that fires off requests to Hadoop (perhaps via JMS, but ...


Only top voted, non community-wiki answers of a minimum length are eligible