3

I am trying to implement a virtual file system structure in front of an object storage (Openstack). For availability reasons we initially chose Cassandra, however while designing file system data model, it looked like a tree structure similar to a relational model. Here is the dilemma for availability and partition tolerance we need NoSQL, but our data model is relational.

The intended file system must be able to handle filtered search based on date, name etc. as fast as possible. So what path should i take? Stick to relational with some indexing mechanism backed by 3 rd tools like Apache Solr or dig deeper into NoSQL and find a suitable model and database satisfying the model?

P.S: Currently from NoSQL Cassandra or MongoDB are choices proposed by my colleagues.

Improve this question
4
  • Is a real file system out of the question? You can index it anywhere --just make sure the thing remains in sync with the index.
    – Apoorv
    Dec 4, 2012 at 12:45
  • CAP theorem says you can't have Consistency, Availability and Partitioning Tolerance all at the same time.
    – NoChance
    Dec 4, 2012 at 12:50
  • @Emmand Yes i know we have also a heavy discussion what to pick two out of three.
    – fgakk
    Dec 4, 2012 at 12:57
  • @Monster well the real file system we use is an object file system, so by virtual i meant to represent objects as files. It is not out of question, but that would mean to change the architecture all together :)
    – fgakk
    Dec 4, 2012 at 13:01

1 Answer 1

Reset to default
5

As I could see from your questions is, that you first should think about what you need and what is NoSql made for.

NoSQL - made for huge amount of distributed data. Scales well from performance an amount of data. and depending in the Type of NoSQL system you can just put and get objects very fast, or also do long running jobs on the distributed data, it's not made for fast search/queries.

Search - it's just made for searching references to data fast. Depending on system it scales well with data and performance. its not made for query huge amount of data nor data relations.

RDBMS - it's made to store data which have relations to each other. System scales depending on system itself and based on your data design. means even the fastest rdbms solutions could perform out with wrong data/query design. It's not made for fast search on huge amount of data. And scaling is depending of rdmbs product, it's not a feature by default.

So if you want to search data from a huge amount, choose the system that is made for it: Search Engine.

If you want to store huge amount of data, where data needs to be distributed (because of amount) and performance of getting data should be independent from amount of data: choose a NoSQL System.

If you don't have that much data,but data needs to have relations to each other, then choose a RDBMS, and think well of you data design.

If you need a Search which stored the index distributed, combine a search product with distributed storage (filesystem).

If you need a distributed filesystem, have a look at Apache Hadoop.

If you need a NoSQL System like Google Big Table, which is somehow compareable to a RDBMS, have a look at Apache Hbase: hbase.apache.org or Hypertable.

Cassandra is more like Amazon Dynamo. It's distributed, but more far away from rdbms.
and MongoDB, is more close to LotusNotes. It's a good storage for Documents/Objects.

And MAIN POINT to solve the problem, think solution dependent, means: do NOT think like RDBMS when using a NoSQL System, you need to think in that specific NoSQL System (they are all very different from each other).

Improve this answer
3
  • Thank you for insightful answer.A good summary of cases and corresponding models.
    – fgakk
    Dec 4, 2012 at 14:04
  • Could you be a bit more precise on what you mean by "huge amount of data"? It would make it clearer.
    – Klaim
    Dec 4, 2012 at 19:47
  • huge amount of data means for me: from size more then 500 GB, from amount of entries more then 100.000.000, they are related to each other, so what comes first. and mostly it depends how this values grows. (so in time) 500 GB in 5 years, is not that much, means with that grow rate you may have time to scale / change / optimize your RDBMS, 500 GB in one year, it is huge, you should now better go for a system that scales fast with this amount. so short, huge amount mostly means: system increases more in shorter times (with bad RDBMS Design you will get bad results with much smaler numbers)
    – fmt.Println.MKO
    Dec 4, 2012 at 21:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.