Software Engineering Stack Exchange is a question and answer site for professionals, academics, and students working within the systems development life cycle. It only takes a minute to sign up.
Sign up to join this community
I am doing an internship on Big Data technologies so I am new to this area. My question is about the use of NoSQL in the Big Data architecture. Do we need always to use a distributed storage (like HDFS in the case of Hadoop) then to put on top a NoSQL databases (like Hbase )?
I find it difficult to understand the typical BIG Data architecture specially for unstructured data .
If you can help me to see better I will be so thankful.
Most big data technologies are distributed by design, because the idea of big data is that your database is too big and too frequented to be handled by one server alone.
But most distributed new database technologies (some of which might be called NoSQL) do not use a separate plattform like HDFS as distributed storage backend but come with their own. MongoDB, to name just one example, runs as different processes on multiple servers which communicate with each other to form clusters and shards.
From http://en.wikipedia.org/wiki/NoSQL
NoSQL database systems are often highly optimized for retrieval and appending operations and often offer little functionality beyond record storage (e.g. key–value stores). The reduced run-time flexibility compared to full SQL systems is compensated by marked gains in scalability and performance for certain data models.
In short, NoSQL database management systems are useful when working with a huge quantity of data when the data's nature does not require a relational model. The data can be structured, but NoSQL is used when what really matters is the ability to store and retrieve great quantities of data, not the relationships between the elements.
For large scale it is better to use distributed storage with NoSQL. But for small scale you can use ordinary storage system.
From http://en.wikipedia.org/wiki/Unstructured_data
Unstructured Data (or unstructured information) refers to information that either does not have a pre-defined data model and/or does not fit well into relational tables. Unstructured information is typically text-heavy, but may contain data such as dates, numbers, and facts as well. This results in irregularities and ambiguities that make it difficult to understand using traditional computer programs as compared to data stored in fielded form in databases or annotated (semantically tagged) in documents.
From http://en.wikipedia.org/wiki/Big_data
In information technology, big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.
Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set. With this difficulty, new platforms of "big data" tools are being developed to handle various aspects of large quantities of data.
To understand BIG Data architecture specially for unstructured data, have a look how Giants work with Big Data.
For example, Google
For example, IBM
For example, Facebook
Distributed storage is an implementation detail more than a necessary thing to understand to handle BIG data. I would focus on grasping unstructured data first, which is more of a feature and more important to grok.
Using Distributed storage is helping to get the proper utilization of NoSQl technologies, if you need to utilize or take the full advantage of these technologies, with big data you must have to use distributed storage.
