28
votes
Microservices and data storage
Of your three options, the first (a single, shared database) and the third (a "database service") are the most common.
The first is called an integration database. This is generally not seen as a ...
- 84.2k
6
votes
What technical approach does NoSQL databases use for fast key-value access
The first big difference is that key-value stores require the user to have the entire key for lookup, and can only be looked up by the key. Contrast that with relational databases which can typically ...
- 148k
6
votes
Accepted
Sharding rule updating on DB scaling
Starting from V11,
Postgres supports Declarative Partitioning where you can divide the table into partitions with built-in Hash partitioning strategy (modulus or round-robin).
It also allows you to ...
- 3,242
4
votes
Accepted
How sharding works when you outgrow the current shard architecture?
One approach that works independent on what sharding rules you use can be following: Use double checks during resharding.
After you added new nodes, start resharding, one node at a time. The node ...
- 864
3
votes
Accepted
Is consistent hashing required for sharding?
Consider sharding as a form of distributed hash table, or distributed range table.
Now it depends on which one the above the sharding is doing.
For a distributed hash table, for each new piece of ...
- 16.3k
3
votes
Modulo Division of Hash Output
Hashes are numbers. Because they are so large, and a fixed number of bits, they are usually written in hexadecimal, but most libraries will also output hashes as a byte array. Byte arrays can be ...
- 148k
3
votes
When to partition a database
SLA
Service Level Agreement.
Are you meeting them?
Hardware
Does your server have the capacity to store the database?
Does it have the (real and potential) capacity to store the expected (and even ...
- 16.3k
3
votes
Accepted
URL shortening : data partitioning
When dealing with a distributed system, you have to accept the fact that, at some point, your nodes will see different data, either because of network partitions or latency over the network. Once you ...
- 3,640
3
votes
Accepted
MongoDB capped collections
What's the potential use for capped collections? (besides logging)
Capped collections are useful for niche use cases where:
You want to limit the amount of data retained in a FIFO (First In, First ...
- 238
2
votes
Accepted
What is the query router in mongodb architecture of sharding?
It means applications don't have to care about what machines the shards are located on. You have to know where at least one of the query routers are because that's what you tell your driver in order ...
- 148k
2
votes
Autosuggest at scale - trie sharding
Suppose you have stored prefix in different instances:
A-F
G-M
N-R
S
T-Z
if there are too many records under A-F instance, you can just re-partition and split it to A-D, E-F.
if there are too many ...
- 121
2
votes
Sharding and application routing cross region on AWS
Main grey area, what can I do when the route goes to the wrong Region/Db?
Re-rout them to the right DB. Seems simple. But...
Tenant 1 went to USA for a holiday and tried to access www.domain.com, ...
- 115k
2
votes
Facebook sharding by user_id or by photo_id?
Engineering doesn't scale evenly. The problems faced when designing/building a single family home are different from those for a typical office building, and are different again for record breaking ...
- 13.8k
2
votes
Accepted
Modulo Division of Hash Output
My question is what method would be used to map the hash output to numeric values so that modulo division can be used?
There are many different ways of converting a string to a number. For example, ...
- 104k
1
vote
Start small - but design in such way that sharding is possible - how?
Option 3 lets you do something the others can't, which I'm going to call "hard sharding": requests are distributed to different servers in different datacenters based entirely on the client'...
- 14.8k
1
vote
URL shortening : data partitioning
One option is to include a shard id in the tiny url you generate. E.g. Let the first n characters of the tiny url represent the shard id. Then when a read request comes in, your load balancer parses ...
1
vote
Merging aggregates with Event sourcing
Event sourcing is not business logic
Your problem description is a bit all over the place. At times you discuss a merge of quizzes, other times you seem to be discussing a combination. I suspect you'...
- 56.3k
1
vote
what extra benefit do I get from a caching layer over the caching layer of the database?
A few things I can think of
If the data is relatively static (think daily or hourly ETL) the cache can save the cost of repeating a slow query in between data changes.
Database servers are expensive. ...
- 903
1
vote
How to use strict schema with seemingly fluid data type
It should be noted that there is no such thing as "flexible data structure". Your code is written against some factual data structure. If you used to use only one email per user, and then, in any way, ...
- 2,585
1
vote
How to use strict schema with seemingly fluid data type
Keep the email addresses in a separate address table.
CREATE TABLE schemaname.address
(
address_id bigserial not null primary key,
user_id bigint not null,
address_type varchar(30) not null,...
- 111
1
vote
Multiple Databases per Microservice
You need to implement horizontal scaling for your whole database, not just the JSON parts. Extracting the JSON parts will only buy you 70% more space once, so you will have the same problem again soon ...
- 61k
1
vote
Multiple Databases per Microservice
Adding a second database isn't going to solve your problem. It will give you a whole bunch of new problems that you've identified, and likely more. What you need to do is structure your data better. ...
- 13.4k
1
vote
Microservices and data storage
Microservices are a form of Service Oriented Architecture, perhaps in the extreme. Their general purpose is to reduce coupling and allow for independent development & deployment.
Very ...
- 34.4k
1
vote
Microservices and data storage
Each microservice has it's own database service - this seems the most
promising, as it preserves the benefits of loose coupling and
horizontal scaling (using redundant database copies and/or ...
- 2,476
1
vote
What technical approach does NoSQL databases use for fast key-value access
If you're just looking at NoSQL generally, versus confined to key-value stores, the possible answers increase.
Certainly with BaseX, and I'm sure other XML, or, even some JSON based databases, XQuery ...
- 224
Only top scored, non community-wiki answers of a minimum length are eligible