Questions tagged [sharding]
strategy for distributing data on several server-nodes that all store the same kind of data. The distribution is based on the content/value of the data. Not to be confused with sharing.
40 questions
4
votes
0
answers
162
views
How do you shard a graph database or graph data?
Imagine I have graph data that is beyond the size of a single machine.
How would you shard a graph database?
I asked on Hacker News and people suggested sharding based on a hash of the predicate-...
- 149
0
votes
1
answer
220
views
How sharding works when you outgrow the current shard architecture?
Looking at sharding techniques, you basically have hash-based or range-based versions. Hash based is more random, range-based is more heuristic based. Say you initially have 2 shards (separate ...
- 2,661
3
votes
2
answers
1k
views
Modulo Division of Hash Output
In a design book I was reading they describe a method to determine a database sharding scheme by taking the hash (MD5, SHA1, whatever) of a userid (integers or uuids) and then (whether encoded or not) ...
- 141
1
vote
1
answer
116
views
When to partition a database
I am reading up on system design concepts, specifically DataBase replication and partitioning. I understand that partitioning is used to improve throughput, and/or when the amount of data is high i.e. ...
- 244
0
votes
2
answers
123
views
Start small - but design in such way that sharding is possible - how?
Following question is more about best-practices than a real problem - nevertheless, I'd like to know how to do it in best way.
Given a service, that can operate in multiple countries/geo-areas, one ...
3
votes
1
answer
1k
views
Is consistent hashing required for sharding?
I am reading about scaling of database and came to know about sharding technique. But I also read about consistent hashing technique. So how practically sharding is implemented? Do we arrange nodes in ...
- 321
2
votes
1
answer
138
views
Sharding rule updating on DB scaling
When you have sharding, horizontal partitioning, you have multiple DBMS instances, and your server app chooses which one to use by some rule that uses record IDs. Well, for example, we have ...
- 191
1
vote
2
answers
460
views
URL shortening : data partitioning
I've been looking in many places about this question - most top Google search results are copy-pasted from a single source, and others are not particularly helpful. Not sure if I am allowed to include ...
- 121
-1
votes
2
answers
1k
views
Merging aggregates with Event sourcing
I'm currently evaluating Event Sourcing and CQRS for an implementation of a new business requirement at my day job. While I can't really speak about the actual business problem, I can give a few ...
- 107
-1
votes
1
answer
404
views
Facebook sharding by user_id or by photo_id?
I'm reading a design suggestion for facebook/instagram social network in grokking the system interview (closed content :( so I'll describe the relevant part in my question).
They are speculating over ...
- 123
0
votes
1
answer
86
views
what extra benefit do I get from a caching layer over the caching layer of the database?
I'm looking at the design of dropbox's edgestore.
It manages several thousands of MySql instances, where all dropbox's metadata resides (users, filenames, etc.). I understand why sharding is ...
- 123
0
votes
2
answers
108
views
How to use strict schema with seemingly fluid data type
Our company is trying to find a good generic way to have Many-to-One data for an entity. For example, a user might have 1 primary email, but many other emails also attached to their account.
So we ...
- 277
3
votes
3
answers
2k
views
Multiple Databases per Microservice
We have a scenario in which all the important and transactional fields of our business entities are highly structured and relational. The data size of these important fields is also very small. ...
- 139
1
vote
0
answers
37
views
Best way to spread/shard file location on a network UNC
We are developing a system whereby documents/files will be stored on a specialized Content Server and uploaded via a client.
However we want to be able to develop this so if we need to, we can swap ...
- 111
0
votes
0
answers
31
views
Source control Sql Server multiple shards with minimal differences
One of the database systems I work with (I'll call it database A) was essentially sharded into 3 schema-identical copies. This was easy to source control, and when a change was made to any of the ...
6
votes
1
answer
1k
views
Autosuggest at scale - trie sharding
While reading on the design for autosuggest implementation on large scale systems (like google), I'm able to understand the usage of trie and how top "n" terms are stored at each node to quickly ...
- 69
5
votes
1
answer
913
views
Sharding and application routing cross region on AWS
We current have a very simple Multi Tenant monolith, with a SQL SERVER backend (Self hosted on EC2 on AWS), and multiple application services talking to one DB behind an Classic AWS ELB. Our database ...
- 101
1
vote
0
answers
850
views
In a distributed system, how to prevent duplicated services from acting multiple times on a common data source?
We are currently rewriting our infrastructure from a monolith to a distributed system because the old system didn`t fit the workload anymore.
In one part of our distributed system, we have an Entity (...
- 191
32
votes
4
answers
12k
views
Microservices and data storage
I'm considering moving a monolithic REST API to a microservice architecture, and I'm getting a bit confused about data storage.
As I see it, some of the benefits of microservices would be:
...
- 429
3
votes
1
answer
3k
views
What is the query router in mongodb architecture of sharding?
I have read the guide and they say
Query Router Sharding is transparent to applications; whether there is
one or one hundred shards, the application code for querying MongoDB
is the same. ...
- 133
1
vote
1
answer
1k
views
MongoDB capped collections
I'm not really sure if that is right "stack" to ask that question, well two questions actually.
What's the potential use for capped collections? (besides logging)
Capped collections cannot be sharded,...
- 113
6
votes
2
answers
854
views
What technical approach does NoSQL databases use for fast key-value access
For example,
In NoSQL, technically replication and sharding approach is used for supporting large data.
Was reading this article about NoSQL use cases. It mentions that NoSQL can be used for ...
- 189
-1
votes
2
answers
443
views
The best shard key for user authentication
For access to my API user should send login and password and get generated token for access.
Account
login
password
Suppose the size of account's table is very large. So large - so needs sharding. ...
- 101
1
vote
1
answer
122
views
Reduce Operations on Distributed Databases
I would like to ask you about optimizing reduce operations (e.g. count) on multiple databases.
Let's say there is a user table that is stored in various countries and sharding is made based on the ...
- 113
0
votes
1
answer
461
views
Building a serverless p2p CMS/System
I'm thinking about building an Open Source, serverless, offline-replicated p2p CMS, but I'm concerned about it really working in a real environment. How would you go about doing that, reducing the ...
user123552
4
votes
0
answers
406
views
How to deal with runtime changes to tenant-location in a clustered, multitenant web application with app-managed datasources?
I have a Java web application that supports multi-tenancy to keep customer data
separate.
Connection pools to each customer database are created at runtime. The
details of each customer shard (...
- 41
1
vote
1
answer
400
views
pre aggregating documents mongodb
In our current setup, most of our database calls are updates where we increment a key by 1.
If all one is doing is incrementing certain keys on a document (a lot) in mongodb, and there are several ...
- 297
3
votes
2
answers
988
views
Thinking pragmatically about scalability
I was recently asked as an exercise to design a scalable graph.
My first intuition was how to seperate the graph and distribute it (sharding,consistent hashing..etc)
Turns out my thinking was on ...
- 133
2
votes
1
answer
2k
views
A multi tenant data architecture
We are at a planning & redesign stage of our loyalty card application. As it is understood it will be a multi-tenant application. And I need to take your thoughts. Here are the some key points we ...
erman olca
6
votes
1
answer
505
views
How to design data structures for sharding?
When designing data structures for an application, what are some best practices to make sure it can be sharded?
In particular, which of the following is better:
Creating relationships in databases
...
2
votes
2
answers
327
views
How can I distribute a unique database already in production?
Let's assume a successful web Spring application running on a MySQL or PostgreSQL database. The traffic is becoming so high and the amount of data is becoming so big that a distributed database ...
3
votes
2
answers
1k
views
multitenancy with some data sharing
I'm in the planning stages of a new webapp, and I am leaning strongly toward a multitenancy model. The app has a file storage function, where the user can upload (and operate on) files.
I would like ...
- 131
3
votes
1
answer
782
views
What is sharding in MongoDB?
I came across sharding in MongoDB but did not get what exactly is it. What I understood is it's better to have more small servers than a huge one for data storage and when the data exceeds the ...
2
votes
1
answer
321
views
Sharding with IoC
I've come across a situation where I need to shard a database (Oracle, but that doesn't particularly matter).
The gist of the problem is I have written a large-scale system in a fairly standard TDD-...
- 3,489
0
votes
1
answer
306
views
Which of the following data duplication options across shards is recommended?
High performance mysql book suggests that for sharding a blog application, one may want to put comments data across 2 shards: first, on the shard of a person posting comment, and on the shard where ...
- 329
2
votes
3
answers
688
views
Sharding and Cloud computing [closed]
I've read that Facebook uses this technique DB sharding to manage its data volume and that the idea of cloud computing is elastic resources. So I'm wondering does the cloud instance take care of ...
- 1,028
3
votes
5
answers
2k
views
what is the benefit of using key/value stores with respect to database sharding?
What has key/value stores got to do with making database sharding easier?
Because if I do not use a key/value store, I can easily shard my database too right?
(Like say, I can easily say users with ...
- 31
11
votes
7
answers
5k
views
Best way for a technical manager to stay up to date on technology
My manager asked for a list of technical blogs he should follow to stay current on technology. His problem is he keeps hearing terms that he hasn't heard of (i.e. NoSql, sharding, agure, sevice bus, ...
- 7,121
2
votes
5
answers
7k
views
English to French translation of computing terminology [closed]
I work in France as a Java programmer, mainly in French, but am a native English speaker. My level of French is pretty good (French wife!), but one thing I have problems with is working out whether to ...
- 315
58
votes
2
answers
81k
views
How would you implement Google Search? [closed]
Supposed you were asked in an interview "How would you implement Google Search?"
How would you answer such a question? There might be resources out there that explain how some pieces in Google are ...
- 1,124