Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.
178

It's not about NoSQL vs SQL, it's about BASE vs ACID. Scalable has to be broken down into its constituents: Read scaling = handle higher volumes of read operations Write scaling = handle higher volumes of write operations ACID-compliant databases (like traditional RDBMS's) can scale reads. They are not inherently less efficient than NoSQL databases ...


80

noSQL databases give up a massive amount of functionality that a SQL database gives you by it's very nature. Things like automatic enforcement of referential integrity, transactions, etc. These are all things that are very handy to have for some problems, and which require some interesting techniques to scale outside of a single server (think about what ...


37

Wow, this is a simple question, which a huge array of possible answers. The more explicit part of your question asks whether it is more scalable to interface with your database directly or through a web service. That answer is simple: query the database directly. Going through the web service adds a whole bunch of latency that is completely unnecessary for ...


27

However, if multiple users are being served by separate servers and both try to add the same item to their cart, for which there is only one remaining, there must be some "source of truth" for the quantity left for that item. Not really. This is not a problem that requires a 100% perfect technical solution, because both error cases have a business solution ...


23

By introducing more servers, you are introducing more lines of communication and the need to keep things synchronized. This is not trivial. The amount of communications overhead goes up quadratically with the number of nodes communicating. If you try to centralize communications you then introduce a scalability bottleneck with communications. There are ...


23

I recommend reading the official answer to your question, Appropriate Uses For SQLite. Specifically, the "Situations Where Another RDBMS May Work Better" warns that SQLite does not support concurrent writing: SQLite supports an unlimited number of simultaneous readers, but it will only allow one writer at any instant in time. For many situations, ...


22

It's actually quite an easy choice. Right now, you have zero users, and scalability is not a problem. Ideally, you want to reach the point where you have millions of users, and scalability becomes a problem. Right now, you don't have a scalability problem; you have a number-of-users problem. If you work on the scalability problem, you will not fix the ...


22

Every time you notice something like that, enter a new ticket into your issue tracking system. Make a habit to use issue tracker as a primary tool to communicate stuff like that, because from there, it will be easy to pick, evaluate and prioritize for your senior colleagues / lead / manager / whoever is responsible for tracking the issues in your project. ...


20

Of course any CPU bound work is going to utilize the CPU. It's going to block the CPU in whatever language or framework you write it in. Node.js is great for when you have I/O bound work, not CPU bound. I wouldn't do heavy lifting in Node, though it can be done. Node.js solves real problems, not fictional or imagined ones like fibonacci number servers. It's ...


20

Some services remove accounts that have not seen any activity in a certain amount of time, say, a year. Others don't bother, on the ground that keeping a user record in their system is a trivial amount of data and who knows, they may come back. Of course if you're keeping track of what users actually do with your service removing them is rather tricky. ...


18

Weirdly, Facebook or Google have so many users that this isn't much of a problem for them. Whoever picked a really desirable username (e.g. "Frank") probably already did so back in 2008. The many, many users who now come and want to try it, never to come back, will probably have to be content with "Frank32183" instead, and once you accept that, there is no ...


16

Well, of course it does. And Java reuses all the infrastructure provided by the JVM, which was written in C, and it ultimately runs in machine code! The point of enabling developers to let their applications scale is not to provide more raw power. Assembler already has all the power that your hardware allows, and no language can provide more. In fact, the ...


16

Start by separating out your functional and non-functional requirements, and carefully defining them. It's very easy to over-design a system based on ambiguous requirements. For Example, your non-functional requirements in regards to scalability is quite ambiguous. It can relieve a lot of mental load if you put actual numbers on your system. For example, ...


16

No, that's not it at all. What you describe is gaining an advantage either by caching (having computed the answer before the request arrived) or by parallelization (tasking more than one node with the computation of a big sum). Neither is necessarily exclusive to 'NoSQL' data bases. (I use scare quotes because what people call 'NoSQL' these days is mostly ...


14

In short: This is a hard problem. Don't reinvent the wheel. There are many technologies that solve the message queue layer. They include ZeroMQ RabbitMQ Apache Kafka Redis, with BLPOP or PUBSUB (I've asked how to do this here). Other AMQP implementations besides RabbitMQ I think it's out of scope for me to discuss the drawbacks of each, not the least ...


14

Answering my own question (aggregating everything I've read so far): Separation of Concerns Implement the business logic in the application in order to benefit from the richer development, testing and debugging environment offered outside the database. Data integrity and cross-cutting concerns (e.g. auditing, reporting) should reside in the database. ...


14

Personally, I'd go with option 3 because: It is normalized and simple Easy to query for reports Easy to back up(just 1 database to worry about) If you index the table well, performance shouldn't be an issue Also, Performance aside, here are some reasons why you would want to avoid option 1 and 2. Cons if you go with 500 databases, 1 per each customer: ...


13

You're trying to denormalise separate facts into a single record, rather than storing them as individual records. Don't. You appear to believe that it will be easier and more efficient to do parsing of pseudo-sets yourself than to store them in the normal way. That is possible but unlikely unless you actually tried it out and determined that your database ...


13

For websites that need to be highly scalable such as social networks like facebook, whats the best way to design the website? Measure. I would think the... Bad policy. Actual measurement is required.


13

How big a data? There are two significant thresholds: whole data fits in the RAM whole index data fits in the RAM With fast SSDs the first threshold became bit less of an issue, unless you have crazy high traffic. ACIDity One of the problem with scaling RDBMSes is that by design they are ACID, which means transactions and row level locks (or even table ...


13

I don't think that the size of data is the only factor. "Data model" is also a very important part. E-Commerce catalog pages (Solr, ElasticSearch), web analytics data (Riak, Cassandra), stock prices (Redis), relationships connections in Social Networks (Neo4J, FleetDB) are just some examples when a NoSQL solution really shines. IMHO, data model has more ...


12

Answer: It doesn't slow down, it does scale up with # of CPU cores. The project used in the original question was 'too small' (it's actually a ton of development but small/optimized for a compiler) to reap the benefits of multiple cores. Seems instead of planning how to spread the work, spawning multiple compiler processes etc, at this small scale it's best ...


11

I would add to that one very common thing - optimizing in the wrong place. I have seen tons of articles around that discuss nanosecond differences in PHP syntax constructs but much less that discuss how to properly design caching infrastructure for an application. So as it was already noted, test. But not just test - profile and find out what exactly is slow ...


11

Actually its tough, but I am sure in lots of comparable situations it is primarily an organizational problem. The only viable approach is probably a mixture of combined measures, not just "one silver bullet". Some things you can try: logging: as I wrote already in a comment, excessive time and resource logging (which is a kind of profiling) can help you to ...


11

My guess is that you need to explore more carefully an approach that you have rejected Enqueue the events on our server My suggestion would be to start reading through the various articles published about the LMAX architecture. They managed to make high volume batching work for their use case, and it may be possible to make your trade offs look more like ...


10

However I have no idea how it will scale if a few people start connecting at once. Premature optimization is the root of all evil. If it works, don't fix it, and if you are curious if it scales then don't lose sleep over it and just test it. It doesn't matter if you'll ever release the software, there are numerous tools that can help you run automated ...


10

What is missing in your system is the cache. You say: However, this requires a lot of separate GetUser calls, causing a lot of separate sql queries inside the User subsystem. The number of calls to a method doesn't have to be the same as the number of SQL queries. You get the information about the user once, why would you query for the same information ...


9

Scalablility is not a function of specific implementation strategies but rather of designing your application architecture so that the data access layer can evolve without massive refactoring and rewriting. An important technique in building a system that scales is to understand your high-level data access requirements and build an interface contract around ...


9

First off, database are good and storing lots of rows. That is something that is ideal for a database. If you know you will never need to do more with the data, and you can guarantee that the string size has a maximum, then it won't hurt, just cause some extra parsing cpu cycles. (But don't be surprised at the fact that things aren't always predictable.) ...


9

Multitenancy is a hard problem. Very hard. I've seen a lot of companies do it, very few of them having done it well. To cite just one of the many examples of where multi-tenant systems can fall apart, consider what will happen when one customers wants a data warehouse or ad-hoc reporting database. Can you spin one off relatively easily, or do you have to go ...


Only top voted, non community-wiki answers of a minimum length are eligible