-3

I have millions of tweets currently stored in HDFS and I plan to analyze them from Spark (Data mining, text mining, Frequent Term-Based Text Clustering, Social Network Analysis) however, do not know if there is any benefit in using a database instead of HDFS for handling data.

There is some justification (in terms of efficiency, workload, etc.) to work with data from any database (perhaps MondoDB) instead of directly into HDFS (stored in json format)? Given that the analysis I will do it from Spark.

1
  • 3
    This is so vague and broad. You need to pay someone to write a twenty-page document analysing your needs and options. And you need to spend a few days with them at the start of that project. Commented Mar 24, 2016 at 2:05

1 Answer 1

0

A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable, partitioned collection of elements that can be operated on in parallel.

Spark work mostly in memory.

As a first answer I will say that to make analytics no need to put the data in database.

2
  • Thanks for your answer... But when I go to extract data to manage with Spark, that process it would not be faster from a database insteed of from HDFS? I mean, in the process in which data is extracted and then become RDD...
    – J Doe
    Commented Mar 24, 2016 at 15:33
  • It depends if it is a distributed database or not. And specialy the need for a NoSql data structure. For raw data usage hdfs will be good. Commented Apr 19, 2016 at 18:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.