QFS (Quantcast File System): Scalable for medium "big data" problems or only for extremely large

Question

I am doing some investigating and research into programming "big data", and am particularly interested in QuantCast (QFS) because it is written in my own favourite programming language, C++.

Looking at how this may play a part in the solution to certain "big data" problems, I know this can manage implementation of peta-byte file systems. I would like to know however how scalable it is for big-data systems that are not as hugely big.

I have seen a lot of comparisons between QFS and Hadoop, and how QFS can outperform Hadoop and so anyone who uses Hadoop may consider using Quantcast (if they just want mega performance).

Essentially is it scalable for medium to large projects? (Someone asked the same about BDD but not Quantcast)

(Also I have seen Hadoop and thus Quantcast being called a "database". I would call it a filesystem, on top of which you might build a database but wouldn't call it a database itself).

Answer 1

DISCLAIMER - I work at Quantcast

The "speed at scale" benefits of QFS certainly become more pronounced the larger the scale of the deployment and the more intense the usage, but that's not to say there isn't some benefit for more modest deployments. However, I would grant that if you aren't running your cluster at full capacity 24/7 (like we are at Quantcast) and your cluster is relatively modest, speed isn't your primary motivator for using QFS.

The benefit that is available at any scale is an increase of logical storage capacity for a given hardware deployment over what HDFS offers for the same level of data integrity due to QFS using Reed-Solomon error correction rather than straight duplication (as in HDFS with 3-way duplication). The QFS approach to error correction also increases the failure tolerance for any given data set from 2 nodes (with 3-way data replication) to 3. More fault tolerant and half the physical space translate in to real dollars when it comes to operating a cluster. The value of this can't be understated, as it basically multiplies the usefulness of your capital expenditure. This financial benefits are the reason why we added the Reed-Solomon feature to QFS.