42

There isn't one; it's a buzzword. The delineator though is that your data is beyond the capabilities of traditional systems. The data is too large to store on the largest disk, the queries take tons too long without special optimization, the network or disk can't support the incoming traffic flow, a plain old dataview isn't going to handle visualization for ...


13

Here's how I see it: Let's ignore the words "big data" for a while, as they are a pretty vague notion You mentioned Hadoop. Hadoop does 2 things: allows you to have a sort of "virtual" drive which is distributed on multiple machines, with redundancy, that can be accessed via Hadoop's API as if it were a single, unitary, drive. It's called HDFS as in Hadoop ...


13

Is it possible to learn these technologies on home PC? Yes. For instance, you can work bith Google AppEngine's SDK entirely offline. Google Code University also provides some starter courses and tutorials on Distributed and Cloud Computing. Which technologies to learn in Cloud Computing? Cloud Computing encompasses many things (ass buzzwords often do.....


11

If you are talking about an application which is bound to stress the limits of the machine, such that you expect you will be doing programming tricks to avoid exceeding those limits, then C++ is the way to go. Not only C++ gives you room for optimization where Java does not, (as Emilio pointed out,) but also, Garbage-Collectors are very memory hungry ...


9

As long as you can run it on a single machine, it's not "Big Data". Your example problem is completely inappropriate to demonstrate anything about it. Big Data means that the problem sizes are so big that distributing the processing is not an optimization but a fundamental requirement. And functional programming makes it much easier to write correct and ...


9

Some alternatives for storing this data: Message queue (possibly distributed), like Apache Kafka This will be optimized for writing and reading a stream of data. It is ideal for collecting data streams in an easy to process format, but it cannot typically be queried except by reading out the stream in its entirety. So, this would be either for archival ...


8

Is Machine Learning a part of Data Science? No. Big Data vs Data Science Not the same. Birds and Bird Watching are also not the same. Machine learning is a type of artificial intelligence (AI) that provides computers with the ability to learn without being explicitly programmed. Machine learning focuses on the development of computer programs that ...


7

The problem is not use C++ as it is Java and not use Java as it is C++. A C++ container is normally implemented to avoid excess of fragmentation, just like the Java free store do. But if you allocate yourself the memory directly, you can do also things Java does not permit you to do, that can result in fragmentation. The proper solution (in C++) is use ...


7

Hadoop was originally written in Java, because it was used to "fix" problems in Nutch, which also was written in Java. Nutch, in turn, was written in Java because it was a write once run anywhere solution. As for whether C++ or another language would have been a better choice, that's definitely up for debate. With modern architectures, I'd trust Java or ...


7

Float Warning Be aware of floating point precision. The mantessa is only x bits long, which means that for values close to zero they can express very small distances, but at galactic distances the error can be quite significant. It also makes a number of logical compression techniques harder. Fixed Point Deltas If you were to go with fixed-point numbers ...


6

I believe most of the "modern" RDBMS implementations are based on the Cascades optimization framework. I shall talk about how Microsoft SQL Server handles this, as that is the DBMS with which I am most familiar. SQL Server is an implementation of the Cascades optimization framework, so it's workings and the ones for other "modern" RDBMS should be similar. ...


6

Without knowing the full details of what you need, you probably want to do one of the following: Use an existing search tool, like Sphinx or Lucene Perform n-gram approximate matching I don't fully know what's involved installing and configuration sphinx; but, I'm under the impression you can point it at a database, tell it which fields to index, how to ...


6

Look into your requirements a little deeper. There is a way to create the illusion of tracking position every second. If you have an app that knows your current GPS location and writes it to a database, why would you keep writing the location if it doesn't change? Even if you require the data, if the user has been asleep for 7 hours, you can ...


5

That is a very vague question, there is no canon definition for what constitutes big data. From a development point of view the only thing that truly changes how you need to handle data is if you have so much that you can't fit it all in memory at once. How much of a problem that is depend greatly on what you need to do with the data, for most jobs you can ...


5

The architecture is good enough to handle many requests per second, as long as you test it and profile it and it proves to handle the load that it is required to handle. Let me quote Donald Knuth, Computer Programming as an Art, 1974: The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at ...


5

OK, so answering your question literally, You don't need to make any changes. The volume of data on the database isn't usually a scaling issue for webpages. Any given webpage will only be looking at a small part of the data and databases are designed to retrieve subsets of data from large sets very quickly. What you need to take account of is how quickly ...


4

Your assumptions are mostly wrong, because you do not account for encodings like UTF-16be. If you take a look at ASCII (or by virtue at UTF-8); the first 32 characters are control characters. This is pretty common for lots of reasons. No. It happens that ASCII, Unicode codepoints and UTF-8 all have control characters at the first 32 byte/codepoint ...


4

Typically I've heard of this being handled via the OLTP vs. OLAP model. Essentially the T in OLTP means "transactional", so this is the typical databased used for day-to-day operations. Then you write some kind of translational logic that transforms the OLTP database into an OLAP database (the A stands for analytical). Basically you're talking about the ...


4

I don't know scala and therefore I cannot comment on your functional approach, but your code looks like overkill. Your recursive function on the other hand is inefficient. Because the function calls itself twice, it is of order 2^n, which is highly inefficient. If you want to compare the three approaches, you need to compare three optimal implementations. ...


4

First, don't assume that a RDMS isn't going to scale. It might not be the right solution, but saying it won't scale doesn't make sense unless you've considered how your data is going to come into the system, how it's going to be queried and what you eventually want to see from those queries. Recording raw page hits may or may not be a large dataset. If ...


4

From http://en.wikipedia.org/wiki/NoSQL NoSQL database systems are often highly optimized for retrieval and appending operations and often offer little functionality beyond record storage (e.g. key–value stores). The reduced run-time flexibility compared to full SQL systems is compensated by marked gains in scalability and performance for certain data ...


4

Use MySQL's INSERT... ON DUPLICATE KEY UPDATE syntax to automatically handle the insert/update logic. 40,000 is not that many rows - I'd be surprised if that command took more than a few seconds. Note that you can insert many rows at once: INSERT INTO table (id, name) VALUES (id1, name1), (id2, name2), ..., (idN, nameN) ON DUPLICATE KEY UPDATE id=VALUES(id)...


4

MapReduce actually has a grouping phase. The map phase essentially consists in transforming inputs into pairs of (key,value) elements. Because the reduce phase consists in "aggregating" all the values associated to the same key, you cannot avoid the need to group all values by key before the reduce phase. This may need a lot of time since values must be ...


4

You're building a training set. This is used to teach the AI what you want. The important thing is to be careful that the set doesn't contain false tells like a red and white checkered table cloth every time it's a pasta dish. We all generalize of course but when humans build training data it's amazingly easy to tip your hand without meaning to. Why ...


3

It depends on what you want from your handling of big data. This concept is relatively vague. For example, if you're talking about MapReduce jobs across disparate data sources, then you may be interested in using Hadoop Streaming with the Dumbo library. If you're talking about statistical analysis, then NumPy and SciPy (as mentioned by Akira71) are ...


3

Aggregated or analytical data is often immutable, that is, it represents a finalized view of data over a certain time period, or w/r/t to some transformational processing. So perhaps some of your problem stems from post hoc alteration of this data. Denormalized data is common in Cassandra, but maybe it would make sense to maintain the individual items (with ...


3

Your project's success is going to depend much more on the features you put in front of the users you manage to attract. For now, I would suggest that you prioritize that. After all, if you don't reach 75M users you won't have a scalability problem anyway, so the effort would be wasted. To phrase this a different way, scalability issues follow from great ...


3

It is difficult to give a simple answer without knowing how the 4 first files are related between them, how the business logic combines data, and if any assumption can be made on ordering of files. Nevertheless, here some general ideas to help you to evaluate yourself the approach you consider. Your data is fixed length, which means easy to parse, compare ...


2

Cloud computing by definition can be learned anywhere. Just signup for one of the free tiers and learn how to start it, expand it, reconfigure it. Discover what prepackaged software and tools can by loaded, setup, used and then discarded. And when you done trash the server and start again.


Only top voted, non community-wiki answers of a minimum length are eligible