Skip to main content

Questions tagged [distributed-system]

This tag can be used by questions on distributed system concept, design, and implementations.

Filter by
Sorted by
Tagged with
0 votes
1 answer
33 views

How to rebalance data across nodes?

I am implementing a message queue where messages are distributed across nodes in a cluster. The goal is to design a system to be able to auto-scale without needing to keep a global map of each message ...
poundifdef's user avatar
1 vote
2 answers
82 views

Subscribing to a range of topics per instance of a service [closed]

If I have a system where my distributed service is sending live scores of 1000 s of football games from some hypothetical event to millions of clients. The service subscribes to the games from a ...
Ufder's user avatar
  • 244
0 votes
1 answer
31 views

how about using kubernetes statefulset to mapping the snowflake datacener id and worker id

I am developing a distribution id project, now using the twitter snowflake id as the fundation of the distribution id. In kubernetes cluster, to fetch the uniq and non-conflict datacenter id and ...
Dolphin's user avatar
  • 101
2 votes
1 answer
229 views

Problem with data consistency in Microservice architecture(2pc transaction) when external dependency envolved

I have microservice architecture. I implement a distributed transaction flow (2-phase commit), kafka, and 1 coordinator service. So, my problem is in each step of preparation I am calling an outside ...
babak abadkheir's user avatar
-2 votes
1 answer
110 views

What is the difference between Eventual Consistency, Strong Eventual Consistency, and Causal Consistency?

I read articles about CRDTs and frequently come across the terms 'Strong Eventual Consistency,' 'Eventual Consistency,' and 'Causal Consistency.' What is the relationship between these consistency ...
Ershetz's user avatar
  • 11
-1 votes
1 answer
71 views

Designing a Distributed System for Indigenous Data Sovereignty Across Nations [closed]

I'm looking for some quick "back-of-the-napkin" thoughts from systems engineers on the following scenario: A collection of Indigenous groups (in the US, Canada, and Australia) want to design ...
Jordan Loewen-Colón's user avatar
6 votes
4 answers
2k views

Protecting against malicious duplicate IDs in a distributed environment

Let's say we have multiple (somewhat autonomous) (micro-)services, and when entities are created, the ID (UUIDs or whatever) can be set externally. How can we ensure that an ID remains unique across ...
Philippe's user avatar
0 votes
0 answers
49 views

Distributed caching system that scales and enables real-time UI updates?

I have a NextJS app for which I want to cache data from our backend APIs to reduce the load when we have high volumes of concurrent users scrolling through (or search) years-worth of historical data. ...
xeno9's user avatar
  • 9
0 votes
1 answer
126 views

Architectural decision regarding at-most-once semantics cron jobs in distributed systems

please note that while I refer to a specific web application framework in the following, the problem also arises with most other web application frameworks I know so please don't be afraid to reply ...
dfsg76's user avatar
  • 109
0 votes
2 answers
179 views

Fault tolerance in aggregated distributed state

I have a scheduling system that is horizontally scaled, and stores shared state in a redis key. The purpose of the system is to implement something similar to classic rate limiting, but a bit ...
NadavShmayo's user avatar
1 vote
2 answers
136 views

Generating unique number in distributed environment

I have little app that i plan to run behind load balancer, with up to 5-8 instances. Inside the app has service, that needs to generate unique number that has up to 10 digits ( so 1-10 digits) each ...
Darlyn's user avatar
  • 209
1 vote
2 answers
99 views

What is the starting point of a geo-distributed server farm?

Scenario: A web application that wants to implement geographically high availability (HA), has multiple web servers in different cities around the globe. Each server is connected to a database in the ...
Parsa99's user avatar
  • 121
1 vote
1 answer
139 views

App that can be deployed both as monolith and microservices?

We have a monolith C++ application and I consider doing some major refactorings. Since we're not completely ready to go full "microservices only" (maybe in about two years time), I want to ...
glades's user avatar
  • 419
0 votes
2 answers
615 views

How to effectively scale one microservice to multiple instances

Let’s say I have a booking microservice (single instance) with its own DB. If I want to horizontally scale the service with multiple instances(exact same copy of the business logic code) running on ...
dig_123's user avatar
  • 137
-2 votes
1 answer
73 views

Rate limiting design

I am designing rate limiting in my application in which the rules can be dynamic. I have a basic design in which every request is validated against the rate limiting quota, which is stored in redis. ...
Sandeep's user avatar
  • 199
4 votes
2 answers
226 views

How do replicas of an app instance assure the task was done only once when the main instance fails?

Let's assume i app have with basic flow: Client send request to server Server do some business logic Server puts message into some message queue Server returns success to the client microservice A ...
Johnyb's user avatar
  • 407
2 votes
3 answers
472 views

How to synchronize cache writes in a distributed system?

I am looking for a complete pattern implementing cache-aside when used in a distributed system across multiple nodes read/writing from/to cache. Specificly how to avoid making multiple requests to db ...
eran otzap's user avatar
1 vote
3 answers
450 views

What algorithm can I use in order to not let there be two rows with same "mobile number" in database?

The issue that I'm facing is on a banking app. There are two ways to register for internet banking. A) Self Register B) Ask for bank to register via their GUI panel backend. Consider this scenario: ...
tuck_y's user avatar
  • 181
-2 votes
1 answer
98 views

SERVICES in sending ENTITIES in distributed systems

Here is a passage from the Blue book: Distributed system architectures, such as J2EE and CORBA, provide special publishing mechanisms for SERVICES , with conventions for their use, and they add ...
cool_cat's user avatar
1 vote
1 answer
367 views

How does a flow of architecture of a digital banking system looks like?

I am working at a digital banking company i.e company that provides software for banks so that the customers of that bank can use banking services via smartphone. But the documentation there is non-...
tuck_y's user avatar
  • 181
-1 votes
3 answers
390 views

Avoid race condition using RDBMS transaction

I am working on design of a distributed system which will process some events. For simplicity lets say, multiple instance of same service will be consuming from same queue, every message will have a ...
Siddhartha Sadhukhan's user avatar
10 votes
3 answers
2k views

How do distributed databases follow unique constraints?

Lets say i have an application where user can register, and the username has to be unqiue value. Now lets say i have N partitions and for each partition i have M replicas with multiple leaders. Now i ...
Johnyb's user avatar
  • 407
1 vote
1 answer
834 views

GRPC over internet

I want to build a grpc server which will be run by a company (service provider) and multiple client each run by a different company (a client). I thought of this instead using a Rest API to get ...
karim alpha's user avatar
3 votes
1 answer
260 views

Best practices for managing software releases with CI/CD with distributed software components running on a customer on-site machine?

I'm looking for some resources on learning how to best manage our software releases in an efficient manner, as our current method slows release schedules down. My current team produces and administers ...
Thomas Smith's user avatar
1 vote
2 answers
78 views

Phantom read race condition keeping RDBMS in sync with external storage

Here's an interesting scenario, consider a cache with many buckets, and resources that can be shared between buckets: Bucket Highest to lowest priority Foo A, B, C, D Bar B, C, D Baz A In the ...
danielrs's user avatar
  • 111
0 votes
2 answers
110 views

How to handle the consumption of multiple event types when timing is crucial?

I have a distributed platform which allows customers to make purchases, and the items which are purchased are stored in an inventory: Sales app -> PurchaseEvent -> Inventory app The Sales app ...
FBryant87's user avatar
  • 323
2 votes
2 answers
102 views

Should a mobile app keep an 'outdated' version of the data and refresh it from database or always fetch from database?

I'm building a mobile, in which the user is able to create,modify and delete entries in a database. There are multiple screen where some of the entries are displayed: (here are some examples, not ...
Corentin Aulagnet's user avatar
0 votes
1 answer
138 views

Redux Pattern in a distributed system?

I would love an opinion on system design in a distributed system that operates under the same domain. I have a bunch of microservices following the Database per Microservice pattern, and updated via ...
user3353167's user avatar
0 votes
1 answer
994 views

Many-To-Many relationships in a Microservice architecture and if the services are correctly identified

I am creating a distributed chat system with a microservice architecture where a user can log in and chat with another user. The services that I identified is a conversation service that handles chat-...
thetruth's user avatar
0 votes
1 answer
115 views

Should I create access control as a microservice or keep the access control logic as part of the specific microservice?

I was wondering if I have a microservice called, for example, Weather, and only certain people can update certain properties around the weather such as Temperature, Precipitation, and Wind. These ...
David King's user avatar
1 vote
1 answer
296 views

Determine when distributed and parallel process is completed

In a distributred system that uses parallel processing to complete tasks and I want to determine when all sub tasks have been completed, what design method or principals can i apply ? I am using ...
Manse's user avatar
  • 149
0 votes
0 answers
37 views

Handling competing requests for the same "id" in different Kubernetes PODs

I have the following problem, I am implementing a delivery management service and a single order can be competed between N deliveryboys, this service is being executed on Kubernetes with 5 PODs of ...
Carlos Rodrigues's user avatar
0 votes
1 answer
296 views

Capture a time-ordered sequence of item-level modifications via events when using Amazon Aurora

I'm designing a small platform based on a series of event-based micro-services. The persistence storage I'm targeting is (the managed) Amazon PostgreSQL (Amazon RDS for PostgreSQL) — although I can go ...
user avatar
0 votes
1 answer
80 views

How to design a fifo coupon distribution system

I have an app written in java and using mysql as DB. I need to create a coupon distributed system where I have 100 coupons stored in DB and the first 100 customers who login should get those coupons. ...
dev Joshi's user avatar
  • 109
3 votes
2 answers
204 views

How to propagate seldomly changing data in a distributed system

We store data – some kind of metadata like car manufacturers or types of computer parts – in one of our applications. This data changes rarely. Let's say it changes once every two weeks. This data is ...
Harold L. Brown's user avatar
3 votes
0 answers
246 views

Microservice architecture but with strong data consistency [closed]

I'm working on a project that has microservice based architecture. Recently we started facing problems where a single user requests requires updating multiple microservices each having it's own ...
JavaLearner's user avatar
0 votes
0 answers
47 views

How to handle multi-region writes for a joint user registration?

UserA and UserB are signing up for some joint service together. They both have to independently agree to T&Cs before sign-up is complete. Once complete they both receive an email to sign-in. A 1 ...
mockitodorito's user avatar
4 votes
3 answers
403 views

How to think about system design questions methodically?

I'm working on my first big system design project at work for the first time in my career. In the past, while i've worked on large systems, I was always working with someone else, so I wasn't really ...
Jeremy Fisher's user avatar
2 votes
1 answer
615 views

Reprocessing out of order events in a stateful stream

Say that you have a monitoring application that reports downtimes for a large number of remote systems (think IOT). A monitoring daemon polls the remote system and reports that status (on/off) via an ...
bangnab's user avatar
  • 746
0 votes
1 answer
55 views

Queue system with priority on Consumers

Imagine a queue system with different consumers - some are low cost whist (like 'reserved/spot' instances in the AWS world) others would be high cost. The system should ensure the messages are ...
Ryan's user avatar
  • 320
0 votes
1 answer
99 views

Operating two independent and competing clusters?

My organization has a redundant setup across 2 data-centers. During normal operations both data-centers are up and handling workloads. For whatever reason, the infrastucture team has decided to ...
LordOfThePigs's user avatar
14 votes
5 answers
7k views

How can I efficiently diff a CSV file against a database?

I have an inventory of products stored in Postgres. I need to be able to take a CSV file and get a list of changes—the things in the CSV file that are different to what is in the database. The CSV ...
Isvara's user avatar
  • 630
0 votes
0 answers
65 views

What is the best way to desgin my api workflow?

I am building a bank management system using the microservices architecture. I have many microservices but I am going to talk about two. I have customer account service and transaction service. Let's ...
ferrocene's user avatar
-2 votes
1 answer
490 views

Is this the right mechanism for 2 way heartbeat?

I have a software application where we have 2 services, say A and B. A delegates some long running work(tracked by an id) to B which is done asynchronously. They both talk via HTTP. Post submitting ...
Saurav Prakash's user avatar
0 votes
1 answer
84 views

How to load consistent test data into multiple applications (which communicate by message streams)?

I have a system landscape consisting of multiple applications. Each application provides REST interfaces to communicate with the outside world. Each application has its own data storage (some kind of ...
Harold L. Brown's user avatar
3 votes
0 answers
92 views

Difficulty understanding how federation increases cache performance for databases?

I am studying system design for distributed systems and in this page (https://github.com/donnemartin/system-design-primer), one of the following advantages was mentioned for federation for databases ...
Henry Zhu's user avatar
  • 153
-1 votes
1 answer
153 views

Checking a related aggregate existence is Domain concern?

This is my specific situation: I am following DDD and Event sourcing to implement my application (using C#). I have a Domain layer that contains Aggregates' implementation. I write unit tests for this ...
Mehrdad's user avatar
  • 99
1 vote
2 answers
238 views

How is cross device eventual consistency achieved in real time?

Assume there is a distributed system which works on the principles of eventual consistency. Let's consider a scenario where I update my profile picture using my phone. If I log in using another system,...
GenZ Dev's user avatar
  • 121
1 vote
1 answer
169 views

Streaming Market Data Over The Public Internet

I'm currently working on a design for a system that can service market data to external clients over the internet. The current solution involves consuming market data from a feed (Feed Handler), this ...
john's user avatar
  • 119
0 votes
1 answer
80 views

designing an order management and dispatch management system for couriers

I'm studying to develop an order management system and order dispatch for the couriers, this is a personal project, for study purposes. Some important points about the deliverymen, they are bikers and ...
Carlos Rodrigues's user avatar

1
2 3 4 5