1

I am traditionally a MsSQL or MySQL developer - I have been prototyping a new application that involves storing data from many targets.

I have seen that the app appears to get slow after scaling to 10k targets after inputting around 300k results for each (64 bit ints) - especially once I start trying to average and/or graph a few targets at once.

Once the app is deployed I will need to add around 1k targets a month and store one data point from each for each and every minute for approx 2 years.

I have never used NoSql, but from reading, it sounds like a good fit - create a document for each target and keep adding fields for each target.

So, I was just wondering, is NoSQL the way to go for this project or should I be sticking with traditional SQL - and/or can anyone offer any advice?

And the second part - If NoSQL is a solution can anyone recommend a fair comparison as I am getting lost in biased reviews that seem to say each solution is superior!?

Improve this question
3
  • 4
    NoSQL is actually not a very helpful term, because it says very little about what something is; all it describes is what it is not. The various NoSQL database systems differ wildly from each other, much more so than the different flavors of SQL databases do.
    – Mason Wheeler
    Mar 5, 2013 at 0:08
  • If by graphing you mean something that traverses graphs and by NoSQL you mean key-value stores, then yes, they're a good match. There are a number of things out there that already do what you appear to be writing from scratch, notably graph databases like Titan.
    – Blrfl
    Mar 5, 2013 at 0:34
  • 2
    Probably depends on how you are storing the charting data. Some opensource JS based graphing libraries love JSON for instance and certain NoSQL DBs also love JSON so depending on what you are doing they may be a perfect match. Without more info though its hard to sell NoSQL vs SQL DBs.
    – Rig
    Mar 5, 2013 at 3:20

2 Answers 2

Reset to default
3

to get the point if either one of the SQL DBs or one kind of NoSQL is right you should first analyze what causes your problem.

is it the amount of data itself, is it the amount of parallel writes, is it the amount of parallel reads, or is it the amount of parallel reads and writes

as far as you have this, you should check how to solve this with SQL, (SQL system are betten then NoSQL systems tells about them)

if you figure out that also a big cluster of SQL with lots of different HD and partitioned data doesn't help then check out depending on your problem which NoSQL systems are there.

as already mentioned there are the graph DBs, they are good for reading and writing lots of connected data. but connections for the data can't be unlimited.

for fast reading there is mongodb, good community and also commercial support.

for fast writing, there is hypertable and hbase . where hypertable is the faster one, but its written in c, and java or php or... apis are only available as thrift libs, which needs a bit more reading how to use it. community is active and responding for both.

mongodb, hypertable and hbase, have all support to create a Map Reduce job. this is usefull if you want todo stuff on your data which takes longer.

means if you dont need a answer to a query (sql speaking) right now, but during the next hour this is the right approach for you.

but for all NoSQL System, you need to start thinking from scratch, so if you start forgot everything from SQL, and only concentrate on the one NoSQL you wanna use. and design your data around this system

one thing that most NoSQL have in common is, that data can be stored twice, on SQL you create foreign keys joins and so. on NoSQL you simple write the data twice (one step to thinking for such systems), (many system event dont have an update method )

Improve this answer
2
  • 1
    What does "(SQL system are betten then NoSQL systems tells about them)" mean?
    – Pete Kirkham
    Mar 5, 2013 at 11:53
  • 3
    sorry for bad english, this means that SQL databases are better then NoSQL databases says about them, or in an example, NoSQL systems says they are the best for big data, they are the fastest and so on, but there are also SQL Databases which can handle a lot of data and they are also very fast, and the they are proven for many years, so before telling that a NoSQL system will solve the problem, you should identify the real problem and figure if and how it can be solved with an SQL database, hope its now more clear what i wanted to say
    – fmt.Println.MKO
    Mar 5, 2013 at 14:12
0

graph databases are sub category of NoSQL. I would recommend you to use a specialized DB for your app. from the description of your app I think any graph/key-value store will do the job. it seems like your have written and tested your app on SQL, moving to noSQl will require you to rewrite/remodel your app. InfiniteGraph, Neo4j and FlockDB are good graph DBs backed my strong community. people from SQL background have complete paradigms shift when working with NOSQL solution, so for the first dive go for the easiest solution. im not the fan boy of any of these, mostly noSQL DBs are free and open source it woldnt take that long to try out few DBs.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.