Skip to main content

Questions tagged [sharding]

strategy for distributing data on several server-nodes that all store the same kind of data. The distribution is based on the content/value of the data. Not to be confused with sharing.

Filter by
Sorted by
Tagged with
4 votes
0 answers
152 views

How do you shard a graph database or graph data?

Imagine I have graph data that is beyond the size of a single machine. How would you shard a graph database? I asked on Hacker News and people suggested sharding based on a hash of the predicate-...
Samuel Squire's user avatar
0 votes
1 answer
178 views

How sharding works when you outgrow the current shard architecture?

Looking at sharding techniques, you basically have hash-based or range-based versions. Hash based is more random, range-based is more heuristic based. Say you initially have 2 shards (separate ...
Lance's user avatar
  • 2,615
2 votes
2 answers
895 views

Modulo Division of Hash Output

In a design book I was reading they describe a method to determine a database sharding scheme by taking the hash (MD5, SHA1, whatever) of a userid (integers or uuids) and then (whether encoded or not) ...
s77's user avatar
  • 131
1 vote
1 answer
105 views

When to partition a database

I am reading up on system design concepts, specifically DataBase replication and partitioning. I understand that partitioning is used to improve throughput, and/or when the amount of data is high i.e. ...
Ufder's user avatar
  • 244
0 votes
2 answers
123 views

Start small - but design in such way that sharding is possible - how?

Following question is more about best-practices than a real problem - nevertheless, I'd like to know how to do it in best way. Given a service, that can operate in multiple countries/geo-areas, one ...
Maciej Pszczolinski's user avatar
3 votes
1 answer
1k views

Is consistent hashing required for sharding?

I am reading about scaling of database and came to know about sharding technique. But I also read about consistent hashing technique. So how practically sharding is implemented? Do we arrange nodes in ...
rahul sharma's user avatar
2 votes
1 answer
135 views

Sharding rule updating on DB scaling

When you have sharding, horizontal partitioning, you have multiple DBMS instances, and your server app chooses which one to use by some rule that uses record IDs. Well, for example, we have ...
AivanF.'s user avatar
  • 191
1 vote
2 answers
409 views

URL shortening : data partitioning

I've been looking in many places about this question - most top Google search results are copy-pasted from a single source, and others are not particularly helpful. Not sure if I am allowed to include ...
Jeeves's user avatar
  • 121
-1 votes
2 answers
970 views

Merging aggregates with Event sourcing

I'm currently evaluating Event Sourcing and CQRS for an implementation of a new business requirement at my day job. While I can't really speak about the actual business problem, I can give a few ...
l7r7's user avatar
  • 107
-1 votes
1 answer
388 views

Facebook sharding by user_id or by photo_id?

I'm reading a design suggestion for facebook/instagram social network in grokking the system interview (closed content :( so I'll describe the relevant part in my question). They are speculating over ...
ihadanny's user avatar
  • 123
0 votes
1 answer
80 views

what extra benefit do I get from a caching layer over the caching layer of the database?

I'm looking at the design of dropbox's edgestore. It manages several thousands of MySql instances, where all dropbox's metadata resides (users, filenames, etc.). I understand why sharding is ...
ihadanny's user avatar
  • 123
0 votes
2 answers
104 views

How to use strict schema with seemingly fluid data type

Our company is trying to find a good generic way to have Many-to-One data for an entity. For example, a user might have 1 primary email, but many other emails also attached to their account. So we ...
Alexander Mills's user avatar
3 votes
3 answers
2k views

Multiple Databases per Microservice

We have a scenario in which all the important and transactional fields of our business entities are highly structured and relational. The data size of these important fields is also very small. ...
iavanish's user avatar
  • 139
1 vote
0 answers
37 views

Best way to spread/shard file location on a network UNC

We are developing a system whereby documents/files will be stored on a specialized Content Server and uploaded via a client. However we want to be able to develop this so if we need to, we can swap ...
user183872's user avatar
0 votes
0 answers
31 views

Source control Sql Server multiple shards with minimal differences

One of the database systems I work with (I'll call it database A) was essentially sharded into 3 schema-identical copies. This was easy to source control, and when a change was made to any of the ...
Jeffrey Van Laethem's user avatar
6 votes
1 answer
999 views

Autosuggest at scale - trie sharding

While reading on the design for autosuggest implementation on large scale systems (like google), I'm able to understand the usage of trie and how top "n" terms are stored at each node to quickly ...
user2599672's user avatar
5 votes
1 answer
876 views

Sharding and application routing cross region on AWS

We current have a very simple Multi Tenant monolith, with a SQL SERVER backend (Self hosted on EC2 on AWS), and multiple application services talking to one DB behind an Classic AWS ELB. Our database ...
Joshscorp's user avatar
  • 101
1 vote
0 answers
796 views

In a distributed system, how to prevent duplicated services from acting multiple times on a common data source?

We are currently rewriting our infrastructure from a monolith to a distributed system because the old system didn`t fit the workload anymore. In one part of our distributed system, we have an Entity (...
looper's user avatar
  • 191
32 votes
4 answers
12k views

Microservices and data storage

I'm considering moving a monolithic REST API to a microservice architecture, and I'm getting a bit confused about data storage. As I see it, some of the benefits of microservices would be: ...
penalosa's user avatar
  • 429
3 votes
1 answer
3k views

What is the query router in mongodb architecture of sharding?

I have read the guide and they say Query Router Sharding is transparent to applications; whether there is one or one hundred shards, the application code for querying MongoDB is the same. ...
Khan's user avatar
  • 133
1 vote
1 answer
1k views

MongoDB capped collections

I'm not really sure if that is right "stack" to ask that question, well two questions actually. What's the potential use for capped collections? (besides logging) Capped collections cannot be sharded,...
GiM's user avatar
  • 113
6 votes
2 answers
845 views

What technical approach does NoSQL databases use for fast key-value access

For example, In NoSQL, technically replication and sharding approach is used for supporting large data. Was reading this article about NoSQL use cases. It mentions that NoSQL can be used for ...
Zameer Ansari's user avatar
-1 votes
2 answers
427 views

The best shard key for user authentication

For access to my API user should send login and password and get generated token for access. Account login password Suppose the size of account's table is very large. So large - so needs sharding. ...
Ivan's user avatar
  • 101
1 vote
1 answer
105 views

Reduce Operations on Distributed Databases

I would like to ask you about optimizing reduce operations (e.g. count) on multiple databases. Let's say there is a user table that is stored in various countries and sharding is made based on the ...
Emre Senturk's user avatar
0 votes
1 answer
455 views

Building a serverless p2p CMS/System

I'm thinking about building an Open Source, serverless, offline-replicated p2p CMS, but I'm concerned about it really working in a real environment. How would you go about doing that, reducing the ...
user avatar
4 votes
0 answers
402 views

How to deal with runtime changes to tenant-location in a clustered, multitenant web application with app-managed datasources?

I have a Java web application that supports multi-tenancy to keep customer data separate. Connection pools to each customer database are created at runtime. The details of each customer shard (...
Jlaud's user avatar
  • 41
1 vote
1 answer
399 views

pre aggregating documents mongodb

In our current setup, most of our database calls are updates where we increment a key by 1. If all one is doing is incrementing certain keys on a document (a lot) in mongodb, and there are several ...
tonyl7126's user avatar
  • 297
3 votes
2 answers
981 views

Thinking pragmatically about scalability

I was recently asked as an exercise to design a scalable graph. My first intuition was how to seperate the graph and distribute it (sharding,consistent hashing..etc) Turns out my thinking was on ...
Stan R.'s user avatar
  • 133
2 votes
1 answer
2k views

A multi tenant data architecture

We are at a planning & redesign stage of our loyalty card application. As it is understood it will be a multi-tenant application. And I need to take your thoughts. Here are the some key points we ...
user avatar
6 votes
1 answer
502 views

How to design data structures for sharding?

When designing data structures for an application, what are some best practices to make sure it can be sharded? In particular, which of the following is better: Creating relationships in databases ...
Jérôme Verstrynge's user avatar
2 votes
2 answers
324 views

How can I distribute a unique database already in production?

Let's assume a successful web Spring application running on a MySQL or PostgreSQL database. The traffic is becoming so high and the amount of data is becoming so big that a distributed database ...
Jérôme Verstrynge's user avatar
3 votes
2 answers
1k views

multitenancy with some data sharing

I'm in the planning stages of a new webapp, and I am leaning strongly toward a multitenancy model. The app has a file storage function, where the user can upload (and operate on) files. I would like ...
meecect's user avatar
  • 131
3 votes
1 answer
781 views

What is sharding in MongoDB?

I came across sharding in MongoDB but did not get what exactly is it. What I understood is it's better to have more small servers than a huge one for data storage and when the data exceeds the ...
chhaya vishwakarma's user avatar
2 votes
1 answer
318 views

Sharding with IoC

I've come across a situation where I need to shard a database (Oracle, but that doesn't particularly matter). The gist of the problem is I have written a large-scale system in a fairly standard TDD-...
Ed James's user avatar
  • 3,499
0 votes
1 answer
302 views

Which of the following data duplication options across shards is recommended?

High performance mysql book suggests that for sharding a blog application, one may want to put comments data across 2 shards: first, on the shard of a person posting comment, and on the shard where ...
jeff musk's user avatar
  • 329
2 votes
3 answers
688 views

Sharding and Cloud computing [closed]

I've read that Facebook uses this technique DB sharding to manage its data volume and that the idea of cloud computing is elastic resources. So I'm wondering does the cloud instance take care of ...
patrick's user avatar
  • 1,028
3 votes
5 answers
2k views

what is the benefit of using key/value stores with respect to database sharding?

What has key/value stores got to do with making database sharding easier? Because if I do not use a key/value store, I can easily shard my database too right? (Like say, I can easily say users with ...
jaytufch's user avatar
11 votes
7 answers
5k views

Best way for a technical manager to stay up to date on technology

My manager asked for a list of technical blogs he should follow to stay current on technology. His problem is he keeps hearing terms that he hasn't heard of (i.e. NoSql, sharding, agure, sevice bus, ...
JoelFan's user avatar
  • 7,091
2 votes
5 answers
7k views

English to French translation of computing terminology [closed]

I work in France as a Java programmer, mainly in French, but am a native English speaker. My level of French is pretty good (French wife!), but one thing I have problems with is working out whether to ...
Rich's user avatar
  • 315
58 votes
2 answers
80k views

How would you implement Google Search? [closed]

Supposed you were asked in an interview "How would you implement Google Search?" How would you answer such a question? There might be resources out there that explain how some pieces in Google are ...
ripper234's user avatar
  • 1,114