211

It's not about NoSQL vs SQL, it's about BASE vs ACID. Scalable has to be broken down into its constituents: Read scaling = handle higher volumes of read operations Write scaling = handle higher volumes of write operations ACID-compliant databases (like traditional RDBMS's) can scale reads. They are not inherently less efficient than NoSQL databases ...


96

noSQL databases give up a massive amount of functionality that a SQL database gives you by it's very nature. Things like automatic enforcement of referential integrity, transactions, etc. These are all things that are very handy to have for some problems, and which require some interesting techniques to scale outside of a single server (think about what ...


27

However, if multiple users are being served by separate servers and both try to add the same item to their cart, for which there is only one remaining, there must be some "source of truth" for the quantity left for that item. Not really. This is not a problem that requires a 100% perfect technical solution, because both error cases have a business solution ...


23

I recommend reading the official answer to your question, Appropriate Uses For SQLite. Specifically, the "Situations Where Another RDBMS May Work Better" warns that SQLite does not support concurrent writing: SQLite supports an unlimited number of simultaneous readers, but it will only allow one writer at any instant in time. For many situations, ...


22

It's actually quite an easy choice. Right now, you have zero users, and scalability is not a problem. Ideally, you want to reach the point where you have millions of users, and scalability becomes a problem. Right now, you don't have a scalability problem; you have a number-of-users problem. If you work on the scalability problem, you will not fix the ...


22

Every time you notice something like that, enter a new ticket into your issue tracking system. Make a habit to use issue tracker as a primary tool to communicate stuff like that, because from there, it will be easy to pick, evaluate and prioritize for your senior colleagues / lead / manager / whoever is responsible for tracking the issues in your project. ...


20

Some services remove accounts that have not seen any activity in a certain amount of time, say, a year. Others don't bother, on the ground that keeping a user record in their system is a trivial amount of data and who knows, they may come back. Of course if you're keeping track of what users actually do with your service removing them is rather tricky. ...


19

Weirdly, Facebook or Google have so many users that this isn't much of a problem for them. Whoever picked a really desirable username (e.g. "Frank") probably already did so back in 2008. The many, many users who now come and want to try it, never to come back, will probably have to be content with "Frank32183" instead, and once you accept that, there is no ...


17

Answering my own question (aggregating everything I've read so far): Separation of Concerns Implement the business logic in the application in order to benefit from the richer development, testing and debugging environment offered outside the database. Data integrity and cross-cutting concerns (e.g. auditing, reporting) should reside in the database. ...


16

In short: This is a hard problem. Don't reinvent the wheel. Many technologies solve the message queue layer. They include ZeroMQ RabbitMQ Apache Kafka Redis, with BLPOP or PUBSUB (I've asked how to do this here). Other AMQP implementations besides RabbitMQ I think it's out of scope for me to discuss the drawbacks of each, not the least because I don't ...


16

Start by separating out your functional and non-functional requirements, and carefully defining them. It's very easy to over-design a system based on ambiguous requirements. For Example, your non-functional requirements in regards to scalability is quite ambiguous. It can relieve a lot of mental load if you put actual numbers on your system. For example, ...


16

No, that's not it at all. What you describe is gaining an advantage either by caching (having computed the answer before the request arrived) or by parallelization (tasking more than one node with the computation of a big sum). Neither is necessarily exclusive to 'NoSQL' data bases. (I use scare quotes because what people call 'NoSQL' these days is mostly ...


15

How big a data? There are two significant thresholds: whole data fits in the RAM whole index data fits in the RAM With fast SSDs the first threshold became bit less of an issue, unless you have crazy high traffic. ACIDity One of the problem with scaling RDBMSes is that by design they are ACID, which means transactions and row level locks (or even table ...


14

Personally, I'd go with option 3 because: It is normalized and simple Easy to query for reports Easy to back up(just 1 database to worry about) If you index the table well, performance shouldn't be an issue Also, Performance aside, here are some reasons why you would want to avoid option 1 and 2. Cons if you go with 500 databases, 1 per each customer: ...


13

I don't think that the size of data is the only factor. "Data model" is also a very important part. E-Commerce catalog pages (Solr, ElasticSearch), web analytics data (Riak, Cassandra), stock prices (Redis), relationships connections in Social Networks (Neo4J, FleetDB) are just some examples when a NoSQL solution really shines. IMHO, data model has more ...


12

Actually its tough, but I am sure in lots of comparable situations it is primarily an organizational problem. The only viable approach is probably a mixture of combined measures, not just "one silver bullet". Some things you can try: logging: as I wrote already in a comment, excessive time and resource logging (which is a kind of profiling) can help you to ...


11

My guess is that you need to explore more carefully an approach that you have rejected Enqueue the events on our server My suggestion would be to start reading through the various articles published about the LMAX architecture. They managed to make high volume batching work for their use case, and it may be possible to make your trade offs look more like ...


10

However I have no idea how it will scale if a few people start connecting at once. Premature optimization is the root of all evil. If it works, don't fix it, and if you are curious if it scales then don't lose sleep over it and just test it. It doesn't matter if you'll ever release the software, there are numerous tools that can help you run automated ...


10

What is missing in your system is the cache. You say: However, this requires a lot of separate GetUser calls, causing a lot of separate sql queries inside the User subsystem. The number of calls to a method doesn't have to be the same as the number of SQL queries. You get the information about the user once, why would you query for the same information ...


10

The basic tradeoff of distributed system is that if you increase the number of write replicas that is needed to be updated for commitment, you improve reliability, but increase latency. On the flip side, if you want to reduce latency, you should reduce the amount of write replicas needed to reach consensus. Distributed databases can be configured in many ...


9

Multitenancy is a hard problem. Very hard. I've seen a lot of companies do it, very few of them having done it well. To cite just one of the many examples of where multi-tenant systems can fall apart, consider what will happen when one customers wants a data warehouse or ad-hoc reporting database. Can you spin one off relatively easily, or do you have to go ...


9

Input stream It is not clear if your 1000 events/second represent peaks or if it's a continuous load: if it's a peak, you could use a message queue as buffer to spread the load on the DB server over a longer time; if it's constant load, the message queue alone is not sufficient, because the DB server will never be able to catch up. Then you'd need to think ...


8

As Robert mentioned in a comment, what they mean is that the software performs well under increasing volumes of load. Any programming language has potential for code that performs or scales well and code that performs or scales poorly. However, Java is an interesting language to be picky about in regards to scaling perhaps because of its popularity and low ...


8

Does using a NoSQL database give a boost to scalability even if you aren't sharding data? Well lets define scalability. If you are referring to scalability as database/backend systems are concerned, in that you have vertical and horizontal scaling where horizontal scaling IS sharding data then this becomes a trivial question because then the answer would be ...


8

For a long time I have been using and recommending the explicit multi-class approach (the second example) as it is the most universal, maintainable, and coherent. Here I will try to support this claim. Single-class This approach has a big problem when there are two or more categories of modifiers. For example a .button with modifiers small, large, huge for ...


8

Optimization is the process of making existing something more efficient, while doing the same operations. This can mean take less processing time for the same operation. It could mean taking less memory. It could mean taking less disk space. Optimization sometimes involves making trade-offs, like using more memory in exchange for faster processing. Making ...


8

What makes me feeling bad about your scenario is exactly what you wrote in the headline and multiple times in the question text: You are the lowest developer in the chain Why is that point so important? Well, first of all, and from a purely technically viewpoint, you are certainly right. You are hired as what you call an "implementor" of things, a worker ...


8

You're a professional. Your employer hired you to be professional. So, treat your concerns the same way you'd want professionals you hire to treat their professional opinions. In particular, you expect other professionals to make necessary optimizations and corrections along the way, provided those optimizations don't unexpectedly increase the cost. For ...


8

You've misread the directionality of the quote. JOINs are generally discouraged for high-volume systems, because they are expensive (because of I/O being necessary more than one table). SO put their database into RAM specifically to avoid being hit by double disk I/O costs, and it turns out that even without physical I/O, the double table searching is ...


8

Separate login credentials from display names Allow users to log in with their email address or existing account from a site which provides such a service (e.g. Google or Facebook). If you really want to have users come up with a new username, that works, too. Then, before interacting with the system further (or as part of registration), ask users to pick ...


Only top voted, non community-wiki answers of a minimum length are eligible