Skip to main content
Commonmark migration
Source Link

Data handling

###Data handling AllAll other things being equal, what is the performance difference between fetching an 8 byte string and a 16 byte string from a database? It is insignificant, and barring some obscure counterexample that is true for any database, SQL or not, the overhead of everything else that goes on in a request dwarfs the time it takes to copy 8 more bytes. If you want to shift data through a database fast then sorting it into big blocks that fit your use case is one of the most significant thing you can do, often way more important than what database software you use.

Data manipulation

###Data manipulation DatabasesDatabases are slow, that is to say, if you implement a data manipulation function in a common program, for instance taking a lot of small strings and joining them into one, and you implement similar functionality through a database request, then the database version will typically take 100 to 1000 times as long to do the operation. The exact figure of course depend on the database, some databases will not be able to do it, so you'd have to write a program that fetch all the data, perform the operation, and then write the result to the database, which is also a pretty slow method.

What database to choose

###What database to choose OnceOnce you have taken all these considerations, what requirements for a database do you have left? Did you manage to come up with a structure that doesn't need any of the fancy/slow features offered by some databases? If you did then an SQL database may be like a Swiss knife with a dull blade, lots of cool features, but not particularly good at what you need it for. Some of the NoSQL databases are simply faster and better when you only need the simple features, others fit the job just as bad as an SQL database.

The big question

###The big question DespiteDespite being written last in this post it is the question you should ask before all the other questions that I mentioned. Do you actually need a database?

###Data handling All other things being equal, what is the performance difference between fetching an 8 byte string and a 16 byte string from a database? It is insignificant, and barring some obscure counterexample that is true for any database, SQL or not, the overhead of everything else that goes on in a request dwarfs the time it takes to copy 8 more bytes. If you want to shift data through a database fast then sorting it into big blocks that fit your use case is one of the most significant thing you can do, often way more important than what database software you use.

###Data manipulation Databases are slow, that is to say, if you implement a data manipulation function in a common program, for instance taking a lot of small strings and joining them into one, and you implement similar functionality through a database request, then the database version will typically take 100 to 1000 times as long to do the operation. The exact figure of course depend on the database, some databases will not be able to do it, so you'd have to write a program that fetch all the data, perform the operation, and then write the result to the database, which is also a pretty slow method.

###What database to choose Once you have taken all these considerations, what requirements for a database do you have left? Did you manage to come up with a structure that doesn't need any of the fancy/slow features offered by some databases? If you did then an SQL database may be like a Swiss knife with a dull blade, lots of cool features, but not particularly good at what you need it for. Some of the NoSQL databases are simply faster and better when you only need the simple features, others fit the job just as bad as an SQL database.

###The big question Despite being written last in this post it is the question you should ask before all the other questions that I mentioned. Do you actually need a database?

Data handling

All other things being equal, what is the performance difference between fetching an 8 byte string and a 16 byte string from a database? It is insignificant, and barring some obscure counterexample that is true for any database, SQL or not, the overhead of everything else that goes on in a request dwarfs the time it takes to copy 8 more bytes. If you want to shift data through a database fast then sorting it into big blocks that fit your use case is one of the most significant thing you can do, often way more important than what database software you use.

Data manipulation

Databases are slow, that is to say, if you implement a data manipulation function in a common program, for instance taking a lot of small strings and joining them into one, and you implement similar functionality through a database request, then the database version will typically take 100 to 1000 times as long to do the operation. The exact figure of course depend on the database, some databases will not be able to do it, so you'd have to write a program that fetch all the data, perform the operation, and then write the result to the database, which is also a pretty slow method.

What database to choose

Once you have taken all these considerations, what requirements for a database do you have left? Did you manage to come up with a structure that doesn't need any of the fancy/slow features offered by some databases? If you did then an SQL database may be like a Swiss knife with a dull blade, lots of cool features, but not particularly good at what you need it for. Some of the NoSQL databases are simply faster and better when you only need the simple features, others fit the job just as bad as an SQL database.

The big question

Despite being written last in this post it is the question you should ask before all the other questions that I mentioned. Do you actually need a database?

Source Link
aaaaaaaaaaaa
  • 1.1k
  • 6
  • 11

First of all a NoSQL database is a database that does not use an SQL interface. What all NoSQL databases have in common is that they do not use an SQL interface. Did I just repeat myself? Yes, but there is nothing else I can say about NoSQL databases as a group. Anything else being said about NoSQL databases around the internet is either wrong for some members of the group, or likely to become so at some point in the future with the release of a new database or a feature upgrade of an existing one.

All this to say that asking if a NoSQL database is a good choice for a particular job is really not an answerable question since different NoSQL databases have hugely different characteristics.

In the scenario that you describe the biggest issue would definitely be that you are pounding Facebook with 8000 HTTP requests per second, but let's ignore that and focus on the quite common issue of having a large amount of tiny data pieces.

###Data handling All other things being equal, what is the performance difference between fetching an 8 byte string and a 16 byte string from a database? It is insignificant, and barring some obscure counterexample that is true for any database, SQL or not, the overhead of everything else that goes on in a request dwarfs the time it takes to copy 8 more bytes. If you want to shift data through a database fast then sorting it into big blocks that fit your use case is one of the most significant thing you can do, often way more important than what database software you use.

Of course there are cases where your use does not fit having big chunks of data, in some cases a caching strategy where data is kept both in the original split form and in chunks may work well, in other cases there is not a lot to do but keeping the small pieces separated.

###Data manipulation Databases are slow, that is to say, if you implement a data manipulation function in a common program, for instance taking a lot of small strings and joining them into one, and you implement similar functionality through a database request, then the database version will typically take 100 to 1000 times as long to do the operation. The exact figure of course depend on the database, some databases will not be able to do it, so you'd have to write a program that fetch all the data, perform the operation, and then write the result to the database, which is also a pretty slow method.

In general, don't do on the database what you could reasonably do to the data before it's written to the database.

###What database to choose Once you have taken all these considerations, what requirements for a database do you have left? Did you manage to come up with a structure that doesn't need any of the fancy/slow features offered by some databases? If you did then an SQL database may be like a Swiss knife with a dull blade, lots of cool features, but not particularly good at what you need it for. Some of the NoSQL databases are simply faster and better when you only need the simple features, others fit the job just as bad as an SQL database.

###The big question Despite being written last in this post it is the question you should ask before all the other questions that I mentioned. Do you actually need a database?

It is a pretty common assumption that when you handle a significant amount of data, you should use a database. But with a modern computer you can store several gigabytes of data in application memory. This gives you fast and easy access, and the good tools for manipulation are right at hand. The one thing that it does not give you is persistence, if the program crash of there is a power loss the data is lost. In a lot of cases that is however perfectly acceptable, your example has data with a life span of ~5 minutes, it doesn't need persistence, it doesn't need a database.