Questions tagged [distributed-system]
This tag can be used by questions on distributed system concept, design, and implementations.
193
questions
0
votes
0
answers
21
views
How to effectively scale one microservice to multiple instances
Let’s say I have a booking microservice (single instance) with its own DB.
If I want to horizontally scale the service with multiple instances(exact same copy of the business logic code) running on ...
- 121
-2
votes
1
answer
54
views
Rate limiting design
I am designing rate limiting in my application in which the rules can be dynamic. I have a basic design in which every request is validated against the rate limiting quota, which is stored in redis. ...
- 99
4
votes
2
answers
154
views
How do replicas of an app instance assure the task was done only once when the main instance fails?
Let's assume i app have with basic flow:
Client send request to server
Server do some business logic
Server puts message into some message queue
Server returns success to the client
microservice A ...
- 397
1
vote
3
answers
158
views
How to synchronize cache writes in a distributed system?
I am looking for a complete pattern implementing cache-aside when used in a distributed system across multiple nodes read/writing from/to cache.
Specificly how to avoid making multiple requests to db ...
- 135
1
vote
3
answers
441
views
What algorithm can I use in order to not let there be two rows with same "mobile number" in database?
The issue that I'm facing is on a banking app. There are two ways to register for internet banking.
A) Self Register
B) Ask for bank to register via their GUI panel backend.
Consider this scenario:
...
- 135
-2
votes
1
answer
96
views
SERVICES in sending ENTITIES in distributed systems
Here is a passage from the Blue book:
Distributed system architectures, such as J2EE and CORBA, provide
special publishing mechanisms for SERVICES , with conventions for
their use, and they add ...
- 37
1
vote
1
answer
206
views
How does a flow of architecture of a digital banking system looks like?
I am working at a digital banking company i.e company that provides software for banks so that the customers of that bank can use banking services via smartphone.
But the documentation there is non-...
- 135
-1
votes
3
answers
193
views
Avoid race condition using RDBMS transaction
I am working on design of a distributed system which will process some events. For simplicity lets say, multiple instance of same service will be consuming from same queue, every message will have a ...
10
votes
3
answers
2k
views
How do distributed databases follow unique constraints?
Lets say i have an application where user can register, and the username has to be unqiue value.
Now lets say i have N partitions and for each partition i have M replicas with multiple leaders.
Now i ...
- 397
1
vote
1
answer
409
views
GRPC over internet
I want to build a grpc server which will be run by a company (service provider) and multiple client each run by a different company (a client). I thought of this instead using a Rest API to get ...
- 13
2
votes
1
answer
99
views
Best practices for managing software releases with CI/CD with distributed software components running on a customer on-site machine?
I'm looking for some resources on learning how to best manage our software releases in an efficient manner, as our current method slows release schedules down.
My current team produces and administers ...
- 37
1
vote
2
answers
76
views
Phantom read race condition keeping RDBMS in sync with external storage
Here's an interesting scenario, consider a cache with many buckets, and resources that can be shared between buckets:
Bucket
Highest to lowest priority
Foo
A, B, C, D
Bar
B, C, D
Baz
A
In the ...
- 111
0
votes
2
answers
91
views
How to handle the consumption of multiple event types when timing is crucial?
I have a distributed platform which allows customers to make purchases, and the items which are purchased are stored in an inventory:
Sales app -> PurchaseEvent -> Inventory app
The Sales app ...
- 323
1
vote
2
answers
84
views
Should a mobile app keep an 'outdated' version of the data and refresh it from database or always fetch from database?
I'm building a mobile, in which the user is able to create,modify and delete entries in a database. There are multiple screen where some of the entries are displayed: (here are some examples, not ...
0
votes
1
answer
103
views
Redux Pattern in a distributed system?
I would love an opinion on system design in a distributed system that operates under the same domain.
I have a bunch of microservices following the Database per Microservice pattern, and updated via ...
- 149
0
votes
1
answer
538
views
Many-To-Many relationships in a Microservice architecture and if the services are correctly identified
I am creating a distributed chat system with a microservice architecture where a user can log in and chat with another user. The services that I identified is a conversation service that handles chat-...
- 11
0
votes
1
answer
75
views
Should I create access control as a microservice or keep the access control logic as part of the specific microservice?
I was wondering if I have a microservice called, for example, Weather, and only certain people can update certain properties around the weather such as Temperature, Precipitation, and Wind.
These ...
1
vote
1
answer
139
views
Determine when distributed and parallel process is completed
In a distributred system that uses parallel processing to complete tasks and I want to determine when all sub tasks have been completed, what design method or principals can i apply ?
I am using ...
- 149
0
votes
0
answers
33
views
Handling competing requests for the same "id" in different Kubernetes PODs
I have the following problem, I am implementing a delivery management service and a single order can be competed between N deliveryboys, this service is being executed on Kubernetes with 5 PODs of ...
0
votes
1
answer
201
views
Capture a time-ordered sequence of item-level modifications via events when using Amazon Aurora
I'm designing a small platform based on a series of event-based micro-services. The persistence storage I'm targeting is (the managed) Amazon PostgreSQL (Amazon RDS for PostgreSQL) — although I can go ...
- 117
0
votes
1
answer
75
views
How to design a fifo coupon distribution system
I have an app written in java and using mysql as DB. I need to create a coupon distributed system where I have 100 coupons stored in DB and the first 100 customers who login should get those coupons. ...
- 109
3
votes
2
answers
196
views
How to propagate seldomly changing data in a distributed system
We store data – some kind of metadata like car manufacturers or types of computer parts – in one of our applications.
This data changes rarely. Let's say it changes once every two weeks.
This data is ...
- 207
3
votes
0
answers
218
views
Microservice architecture but with strong data consistency [closed]
I'm working on a project that has microservice based architecture.
Recently we started facing problems where a single user requests requires updating multiple microservices each having it's own ...
- 139
0
votes
0
answers
46
views
How to handle multi-region writes for a joint user registration?
UserA and UserB are signing up for some joint service together. They both have to independently agree to T&Cs before sign-up is complete. Once complete they both receive an email to sign-in.
A 1 ...
4
votes
3
answers
368
views
How to think about system design questions methodically?
I'm working on my first big system design project at work for the first time in my career. In the past, while i've worked on large systems, I was always working with someone else, so I wasn't really ...
- 327
2
votes
1
answer
369
views
Reprocessing out of order events in a stateful stream
Say that you have a monitoring application that reports downtimes for a large number of remote systems (think IOT). A monitoring daemon polls the remote system and reports that status (on/off) via an ...
- 746
0
votes
1
answer
54
views
Queue system with priority on Consumers
Imagine a queue system with different consumers - some are low cost whist (like 'reserved/spot' instances in the AWS world) others would be high cost.
The system should ensure the messages are ...
- 320
0
votes
1
answer
98
views
Operating two independent and competing clusters?
My organization has a redundant setup across 2 data-centers. During normal operations both data-centers are up and handling workloads. For whatever reason, the infrastucture team has decided to ...
- 1,101
14
votes
5
answers
6k
views
How can I efficiently diff a CSV file against a database?
I have an inventory of products stored in Postgres. I need to be able to take a CSV file and get a list of changes—the things in the CSV file that are different to what is in the database. The CSV ...
- 620
0
votes
0
answers
63
views
What is the best way to desgin my api workflow?
I am building a bank management system using the microservices architecture. I have many microservices but I am going to talk about two. I have customer account service and transaction service. Let's ...
- 17
-2
votes
1
answer
421
views
Is this the right mechanism for 2 way heartbeat?
I have a software application where we have 2 services, say A and B.
A delegates some long running work(tracked by an id) to B which is done asynchronously.
They both talk via HTTP.
Post submitting ...
- 107
0
votes
1
answer
83
views
How to load consistent test data into multiple applications (which communicate by message streams)?
I have a system landscape consisting of multiple applications.
Each application provides REST interfaces to communicate with the outside world.
Each application has its own data storage (some kind of ...
- 207
3
votes
0
answers
91
views
Difficulty understanding how federation increases cache performance for databases?
I am studying system design for distributed systems and in this page (https://github.com/donnemartin/system-design-primer), one of the following advantages was mentioned for federation for databases ...
- 153
-1
votes
1
answer
148
views
Checking a related aggregate existence is Domain concern?
This is my specific situation:
I am following DDD and Event sourcing to implement my application (using C#). I have a Domain layer that contains Aggregates' implementation. I write unit tests for this ...
- 99
1
vote
2
answers
220
views
How is cross device eventual consistency achieved in real time?
Assume there is a distributed system which works on the principles of eventual consistency.
Let's consider a scenario where I update my profile picture using my phone. If I log in using another system,...
- 121
1
vote
1
answer
151
views
Streaming Market Data Over The Public Internet
I'm currently working on a design for a system that can service market data to external clients over the internet. The current solution involves consuming market data from a feed (Feed Handler), this ...
- 119
0
votes
1
answer
65
views
designing an order management and dispatch management system for couriers
I'm studying to develop an order management system and order dispatch for the couriers, this is a personal project, for study purposes.
Some important points about the deliverymen, they are bikers and ...
0
votes
0
answers
464
views
Dealing with third party api rate limit when using multiple services
I have 1 api key that has a rate limit 1000req/1min.
I would like to somehow use 50% of that limit in service 1 and 50% of that limit in service 2. Let's say they're just 2 separate processes using ...
- 1,559
-1
votes
1
answer
111
views
Design approach to decouple DB write to reduce application latency
I want to use DynamoDB to save application operation data at the end of a session for auditing purposes without affecting the actual application latency. Because this date is only for auditing so we ...
0
votes
0
answers
138
views
Load balancing the load balancers
Currently I have a system where I have installed HAProxy on one machine and my other 3 machines serves the webapps and the fourth machine for the database. Now I need to add another load balancer in ...
- 135
3
votes
0
answers
635
views
System Design: Chat application in a distributed system context
I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
- 31
0
votes
1
answer
202
views
Is publishing to the same event store stream from multiple places an anti pattern in a distributed transaction?
Below is a little context to the problem I'm trying to solve.
There are two separate microservices with separate frontends , APIs and databases.
Customer Microservice - Manages customers
Orders ...
- 302
5
votes
1
answer
475
views
Can an api gateway handle websockets requests besides regular http requests at the same time?
I am working on social platform with a distributed architecture. The platform should offer a chat module, but since the most efficient way to implement a chat is using websockets. I am not sure what ...
- 159
-3
votes
1
answer
230
views
How to manage a distributed application?
I have a distributed application backed by a distributed database. (Distributed here means running across multiple hosts with one or more instances on each host). Application has several toolings ...
- 101
3
votes
1
answer
1k
views
Best strategy to flush Redis cache used in a distributed environment
Current setup:
We follow the cache-first strategy right now.
Always fetch from the cache first.
If absent in cache, fetch from DB and update the cache.
Cache TTL is 30 min. After this, the cache-key ...
- 139
6
votes
1
answer
740
views
Should user and service-to-service authentications be separate?
Say I have a system with 5 microservices behind a gateway, and a user signs in through an IDP (OAuth)
A user U passes the access token in a request, and the call first reaches the gateway before it ...
- 171
2
votes
0
answers
199
views
Automatically extended database-based mutex lock - is it a good idea?
Since I am working in a multi-instance microservice environment, I came accross a problem with making some operations being performed at most by one of the running instances at once. The solution ...
- 129
0
votes
0
answers
84
views
Realtime report generation and downloads
We host a dashboard where users can view different reports, each powered by a templated SQL query. The users have an option to "download" those reports as CSV.
Currently the above use case ...
- 101
2
votes
3
answers
1k
views
How can message queues improve scalability?
I have read in a DDD book that using message queues between communicating services can make the whole architecture more scalable, amazon's documentation mentions that queues provide granular ...
- 293
1
vote
1
answer
1k
views
What performance benefits does a Redis cluster have over having all machines connect to a single Redis node? [closed]
Suppose I have data that easily fits in memory in a redis database. The redis server runs as a single service on a single process on a single machine. Let's say I have a 100 different machines connect ...
- 121