6

I found an interesting article on high scalability web site where it talks about EBay scalability and especially a specific passage struck me:

"(Ebay strategies)...Move work out of the database into the applications because the database is the bottleneck. Ebay does this in the extreme. We see it in other architecture using caching and the file system, but eBay even does a lot of traditional database operations in applications (like joins)."

and the above is no mistake because again in the same article:

"Move cpu-intensive work moved out of the database layer to applications applications layer: referential integrity, joins, sorting done in the application layer! Reasoning: app servers are cheap, databases are the bottleneck."

Any explanation on this? If the above were true then I should only use the database for retrieving data and do all other operations in the programming logic.

I was always told the opposite: "databases are optimized for operations on data and complex selects so use them".

Any insight?

Improve this question
0

3 Answers 3

Reset to default
7

"Measure. Don't Guess".

We can't assume that Ebay's bottlenecks are the same as our own. For the specific applications that I work on, when a bottleneck exists it is rarely the database (or if it is, it's because of poorly optimized queries). I know this because we've reviewed the instances of poor-performance and profiled the performance of normal performance of our application.

Improve this answer
1
  • 1
    Ok I agree, we'll never know until we measure, however I'd like a general opinion on the matter. Measuring without knowing what the expected output should be is not useful for me.
    – dendini
    Jul 30, 2013 at 12:02
6

Yes, relational databases are bottlenecks when it comes to horizontal scalability; in order to scale your RDBMS server in most cases you need a more powerful machine and this leads to a limit. This is one of the main reasons why NoSQL databases have appeared as an alternative to RDBMS, by trading off ACID transactions. The bottleneck is even bigger if you put application logic in the database, in the form of stored procedures.

By shifting the processing power in the application tier, your whole application can scale out more easily, since a well written application server can be deployed on multiple machine servers. An alternative to this is using the above mentioned NoSQL data stores for achieving scalability; NoSQL data stores can also be easily deployed onto multiple nodes.

Note, that this only makes sens in a high-scalability talk, as you have pointed out in the question. In most cases, when used properly, databases represent no real bottleneck and do their job very well as they have done it for decades.

Improve this answer
3
  • So putting the logic in the program instead of in SQL queries is a desperate hack to avoid using a proper NoSQL database and scaling horizontally with it? Am I right?
    – dendini
    Jul 30, 2013 at 12:00
  • 2
    @dendini No, putting the application logic in the application is what's usually recommended (too much app logic in the database is highly suspect). Here we're talking about moving query logic out of the database. See the difference? Regardless, these are no "desperate hacks", but trade-offs -- the bread and butter of software development.
    – Andres F.
    Jul 30, 2013 at 12:22
  • 1
    @dendini No; you may want ACID transactions, which highly scalable NoSQL do not provide. Putting logic in SQL query (where, join) is in almost all cases a good idea, since is what relational databases do well; application logic in the database is used in stored procedure and it's debatable whether or not is a good practice (in my opinion it is not, due to poor scalability).
    – Random42
    Jul 30, 2013 at 13:31
5

Memory is fast and non-durable, while drives are slow and durable. If you pull data out of a database and pop it into memory, its access will always be significantly quicker the downside is that if the server fails, you'll lose the data.

Databases are a great fit for most applications out there. Only a small percentage of applications have the scale of google, twitter etc, and while its always fascinating on how those companies have overcome their challenges, it would be naive to try implement their solutions from the word go.

Improve this answer
3
  • 4
    Easy trap to fall into, "Company X does this completely crazy thing, so I should too!!!" Some of these high scalability techniques sound elegant and clever, but I image when you get down to the nitty gritty they are nothing more than desperate hacks.
    – Tombatron
    Jul 30, 2013 at 10:41
  • You seem to intere that servers don't use memory for processing requests.
    – Tulains Córdova
    Jul 30, 2013 at 11:27
  • The question was whether to create complex queries and let the database return the results or just using the database to retrieve data and then do operations on it in the program. In both cases disk or memory could be hit by the RDBMS but it's not the point
    – dendini
    Jul 30, 2013 at 12:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.