Questions tagged [distributed-system]
This tag can be used by questions on distributed system concept, design, and implementations.
180
questions
1
vote
1
answer
62
views
Capture a time-ordered sequence of item-level modifications via events when using Amazon Aurora
I'm designing a small platform based on a series of event-based micro-services. The persistence storage I'm targeting is (the managed) Amazon PostgreSQL (Amazon RDS for PostgreSQL) — although I can go ...
CommunityBot
- 1
0
votes
1
answer
79
views
Many-To-Many relationships in a Microservice architecture and if the services are correctly identified
I am creating a distributed chat system with a microservice architecture where a user can log in and chat with another user. The services that I identified is a conversation service that handles chat-...
- 24.5k
0
votes
1
answer
57
views
Should I create access control as a microservice or keep the access control logic as part of the specific microservice?
I was wondering if I have a microservice called, for example, Weather, and only certain people can update certain properties around the weather such as Temperature, Precipitation, and Wind.
These ...
- 70.2k
0
votes
1
answer
52
views
Determine when distributed and parallel process is completed
In a distributred system that uses parallel processing to complete tasks and I want to determine when all sub tasks have been completed, what design method or principals can i apply ?
I am using ...
- 13.8k
0
votes
1
answer
98
views
Operating two independent and competing clusters?
My organization has a redundant setup across 2 data-centers. During normal operations both data-centers are up and handling workloads. For whatever reason, the infrastucture team has decided to ...
CommunityBot
- 1
0
votes
0
answers
20
views
Handling competing requests for the same "id" in different Kubernetes PODs
I have the following problem, I am implementing a delivery management service and a single order can be competed between N deliveryboys, this service is being executed on Kubernetes with 5 PODs of ...
7
votes
3
answers
3k
views
How to prevent overlap booking on a calendar booking system
As an exercise, I am trying to design a simple calendar booking system for multiple meeting rooms. I kind of got my head around some requirements such as find available rooms for a given time range, ...
- 119
3
votes
3
answers
2k
views
Message Broker in Client-Server-Applications
I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ).
A simplified diagram of my ...
CommunityBot
- 1
0
votes
1
answer
55
views
How to design a fifo coupon distribution system
I have an app written in java and using mysql as DB. I need to create a coupon distributed system where I have 100 coupons stored in DB and the first 100 customers who login should get those coupons. ...
- 194k
1
vote
1
answer
352
views
Processing data in a cluster with multiple tasks
The goal is to provide a scaleable system implementing multiple data processing tasks which can be seen as a graph. Data objects will travel that graph. Most object will visit the same nodes in the ...
CommunityBot
- 1
3
votes
2
answers
184
views
How to propagate seldomly changing data in a distributed system
We store data – some kind of metadata like car manufacturers or types of computer parts – in one of our applications.
This data changes rarely. Let's say it changes once every two weeks.
This data is ...
- 13.3k
0
votes
1
answer
62
views
How to load consistent test data into multiple applications (which communicate by message streams)?
I have a system landscape consisting of multiple applications.
Each application provides REST interfaces to communicate with the outside world.
Each application has its own data storage (some kind of ...
CommunityBot
- 1
4
votes
1
answer
313
views
Can an api gateway handle websockets requests besides regular http requests at the same time?
I am working on social platform with a distributed architecture. The platform should offer a chat module, but since the most efficient way to implement a chat is using websockets. I am not sure what ...
CommunityBot
- 1
0
votes
1
answer
115
views
Is publishing to the same event store stream from multiple places an anti pattern in a distributed transaction?
Below is a little context to the problem I'm trying to solve.
There are two separate microservices with separate frontends , APIs and databases.
Customer Microservice - Manages customers
Orders ...
CommunityBot
- 1
3
votes
1
answer
585
views
What are the different messaging patterns for multiple producers to single consumer through a distributed queue?
We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
CommunityBot
- 1
1
vote
1
answer
392
views
Regarding data modeling in a NoSQL use case
I am reading the following link to learn about system designs of various systems. (This is a paid link, I am attaching all explanations below.)
In an attempt to explain the system design of Instagram ...
CommunityBot
- 1
3
votes
0
answers
189
views
Microservice architecture but with strong data consistency [closed]
I'm working on a project that has microservice based architecture.
Recently we started facing problems where a single user requests requires updating multiple microservices each having it's own ...
- 139
2
votes
1
answer
377
views
Design question on synchronization of two asynchronous data streams
I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
CommunityBot
- 1
1
vote
1
answer
1k
views
RabbitMQ: how to achieve scalable one-to-one communication?
I'm trying to use RabbitMQ for two-way one-to-one communication between clients and servers. So, Client 1 should be able to communicate only with Server 1, and vice versa:
The servers stay always ...
CommunityBot
- 1
2
votes
3
answers
656
views
How can message queues improve scalability?
I have read in a DDD book that using message queues between communicating services can make the whole architecture more scalable, amazon's documentation mentions that queues provide granular ...
- 169
0
votes
0
answers
40
views
How to handle multi-region writes for a joint user registration?
UserA and UserB are signing up for some joint service together. They both have to independently agree to T&Cs before sign-up is complete. Once complete they both receive an email to sign-in.
A 1 ...
4
votes
3
answers
347
views
How to think about system design questions methodically?
I'm working on my first big system design project at work for the first time in my career. In the past, while i've worked on large systems, I was always working with someone else, so I wasn't really ...
- 5,656
2
votes
1
answer
84
views
Reprocessing out of order events in a stateful stream
Say that you have a monitoring application that reports downtimes for a large number of remote systems (think IOT). A monitoring daemon polls the remote system and reports that status (on/off) via an ...
- 251
0
votes
1
answer
45
views
Queue system with priority on Consumers
Imagine a queue system with different consumers - some are low cost whist (like 'reserved/spot' instances in the AWS world) others would be high cost.
The system should ensure the messages are ...
- 124
-2
votes
1
answer
345
views
Is this the right mechanism for 2 way heartbeat?
I have a software application where we have 2 services, say A and B.
A delegates some long running work(tracked by an id) to B which is done asynchronously.
They both talk via HTTP.
Post submitting ...
- 11k
14
votes
5
answers
6k
views
How can I efficiently diff a CSV file against a database?
I have an inventory of products stored in Postgres. I need to be able to take a CSV file and get a list of changes—the things in the CSV file that are different to what is in the database. The CSV ...
- 1,230
-3
votes
2
answers
220
views
How to manage a distributed application?
I have a distributed application backed by a distributed database. (Distributed here means running across multiple hosts with one or more instances on each host). Application has several toolings ...
- 108
0
votes
2
answers
3k
views
Race condition in Distributed Systems
We have a set of micro-services that are functionally dependent and have intertwined validation logic. e.g. consider 'Credit-Card' service and 'Loan' service. When user has a credit card then loan ...
- 24.5k
0
votes
0
answers
61
views
What is the best way to desgin my api workflow?
I am building a bank management system using the microservices architecture. I have many microservices but I am going to talk about two. I have customer account service and transaction service. Let's ...
- 17
0
votes
0
answers
50
views
Cloud solution to parse and process 1M+ rows
We are writing a service that can parse a CSV file containing over 1M+ rows. Each row needs insertion (or update) data of a DB record. We are currently using DynamoDB.
What AWS services are best ...
- 117
-1
votes
1
answer
89
views
Design approach to decouple DB write to reduce application latency
I want to use DynamoDB to save application operation data at the end of a session for auditing purposes without affecting the actual application latency. Because this date is only for auditing so we ...
CommunityBot
- 1
3
votes
0
answers
87
views
Difficulty understanding how federation increases cache performance for databases?
I am studying system design for distributed systems and in this page (https://github.com/donnemartin/system-design-primer), one of the following advantages was mentioned for federation for databases ...
- 153
0
votes
1
answer
174
views
Consistency and Availability in distributed hashing Key value store
During the last semester of my college, we were asked to read the (chord paper) and implement it without fault tolerance. We built it with the following feature:
A new node can join.
A client can ...
- 186
-1
votes
1
answer
140
views
Checking a related aggregate existence is Domain concern?
This is my specific situation:
I am following DDD and Event sourcing to implement my application (using C#). I have a Domain layer that contains Aggregates' implementation. I write unit tests for this ...
CommunityBot
- 1
-4
votes
3
answers
502
views
How does a distributed system both tolerate network partition and achieve consistency?
In the CAP theorem, there are only three possible cases:
C and A without P
C and P without A
A and P without C
How can we have both P and C in the second case?
Doesn't propagation of update from ...
- 186
1
vote
0
answers
259
views
Bi-directional messaging using Zero MQ pub sub pattern
I am trying to design a system that can be safely used to share messages between processes on the same system using ZeroMQ.
I have chosen to use a central broker that binds to a particular port.
All ...
0
votes
2
answers
160
views
How is cross device eventual consistency achieved in real time?
Assume there is a distributed system which works on the principles of eventual consistency.
Let's consider a scenario where I update my profile picture using my phone. If I log in using another system,...
- 130k
1
vote
1
answer
128
views
Streaming Market Data Over The Public Internet
I'm currently working on a design for a system that can service market data to external clients over the internet. The current solution involves consuming market data from a feed (Feed Handler), this ...
- 843
0
votes
1
answer
121
views
Maintaining Objects Across API Deployment Instances
I am working on a web application as a hobby and trying to learn some concepts related to cloud development and distributed applications. I am currently targeting an AWS EC2 instance as a deployment ...
CommunityBot
- 1
0
votes
1
answer
54
views
designing an order management and dispatch management system for couriers
I'm studying to develop an order management system and order dispatch for the couriers, this is a personal project, for study purposes.
Some important points about the deliverymen, they are bikers and ...
- 13.8k
3
votes
5
answers
2k
views
How to resolve cyclic dependencies in Event-driven systems?
Consider two services (bounded contexts by DDD):
Sales
Billing
Sales is responsible for creating orders and Billing for handling payments.
Sales tracks orders and Billing holds payments:
Sales ...
- 335
0
votes
0
answers
293
views
Dealing with third party api rate limit when using multiple services
I have 1 api key that has a rate limit 1000req/1min.
I would like to somehow use 50% of that limit in service 1 and 50% of that limit in service 2. Let's say they're just 2 separate processes using ...
- 1,489
0
votes
0
answers
92
views
Load balancing the load balancers
Currently I have a system where I have installed HAProxy on one machine and my other 3 machines serves the webapps and the fourth machine for the database. Now I need to add another load balancer in ...
- 109
3
votes
0
answers
442
views
System Design: Chat application in a distributed system context
I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
- 31
13
votes
2
answers
689
views
When bounded contexts and "microservices" collide. A distributed systems dilemma in diagram form
You can't please everyone. Some people want a lot of context and background on sites like this. Others do not. If you don't want the background, skip the first three paragraphs.
I am a software ...
- 806
-4
votes
5
answers
966
views
Why can't we vertically scale a system infinitely?
When talking about scaling a system, it's often said that vertical scaling has limitations. So after a point, we need to scale the system horizontally.
What are the limitations of:
Main memory - ...
- 111
-1
votes
1
answer
68
views
How to design a topped-up internal balance for a replicated API?
My product is an API which charges an amount of USD every N requests.
I would like to allow my users to top-up their internal balances through PayPal, then decrease their balances when they use the ...
CommunityBot
- 1
2
votes
2
answers
170
views
Is sequential consistency equivalent to performing memory accesses by a processes in program order and performing each memory access atomically?
In Fundamentals of Parallel Multicore Architecture, by Yan Solihin, p304 defines sequential consistency memory model:
Overall, we can express programmers’ implicit expectation of memory
access ...
- 111
4
votes
2
answers
121
views
Do coherence and transactional isolation mean the same?
About coherence vs consistency, https://en.wikipedia.org/wiki/Consistency_model says
Coherence deals with maintaining a global order in which writes to a
single location or single variable are seen ...
- 111
3
votes
1
answer
957
views
Best strategy to flush Redis cache used in a distributed environment
Current setup:
We follow the cache-first strategy right now.
Always fetch from the cache first.
If absent in cache, fetch from DB and update the cache.
Cache TTL is 30 min. After this, the cache-key ...
- 68.8k