Questions tagged [distributed-system]
This tag can be used by questions on distributed system concept, design, and implementations.
45
questions with no upvoted or accepted answers
4
votes
1
answer
250
views
Can an api gateway handle websockets requests besides regular http requests at the same time?
I am working on social platform with a distributed architecture. The platform should offer a chat module, but since the most efficient way to implement a chat is using websockets. I am not sure what ...
- 149
3
votes
0
answers
83
views
Difficulty understanding how federation increases cache performance for databases?
I am studying system design for distributed systems and in this page (https://github.com/donnemartin/system-design-primer), one of the following advantages was mentioned for federation for databases ...
- 153
3
votes
0
answers
53
views
Release Phase in Lamport's Mutual Exclusion Algorithm
I was studying Lamport's mutual exclusion algorithm from the original paper, and noticed a difference in the release phase to the one given in Wikipedia. In the original paper, author states that :
...
- 181
3
votes
0
answers
66
views
Sharing tasks among parallel independent jobs
I am getting files on HDFS and I have a legacy java app which checks for new files every 15 mins, does some transformations and write them to some remote server.
Now, I am facing scaling issue with ...
- 209
3
votes
0
answers
84
views
Architecture and implementation of a de-centralised web service
I'm a web programmer with several years of experience in RoR and node.js. I have a project idea that is quite big and I need some input in how it can be achieved.
I heard of the Diaspora project, and ...
- 291
3
votes
0
answers
714
views
Vector clock conflict resolution in event sourced system via local timestamp - is this safe?
We have an occasionally connected, event sourced system: a budgeting application where each client can pull other client's event store from a separate storage (network share, dropbox,...).
It is our ...
- 131
3
votes
1
answer
151
views
Database consistency for a distributed system
I'm working on a phone call monitoring project. The aim is to have one row in the calls table for each ongoing call. Each call instance may be updated by many different servers/threads as more ...
- 141
2
votes
0
answers
230
views
System Design: Chat application in a distributed system context
I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
- 21
2
votes
0
answers
90
views
Automatically extended database-based mutex lock - is it a good idea?
Since I am working in a multi-instance microservice environment, I came accross a problem with making some operations being performed at most by one of the running instances at once. The solution ...
- 129
2
votes
0
answers
70
views
SAGA pattern - how to handle situation when some message is not received by orchestrator
Recently I'm studying Saga pattern, but I'm struggling to imagine how it will work in real system. Standard example is making an order, which consist of reserving flight, hotel and a car.
We can ...
2
votes
1
answer
412
views
What are the different messaging patterns for multiple producers to single consumer through a distributed queue?
We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
- 129
2
votes
0
answers
71
views
Distributing tasks to several clients
I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements.
I have 1-...
- 111
2
votes
1
answer
347
views
Design question on synchronization of two asynchronous data streams
I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
2
votes
3
answers
2k
views
Message Broker in Client-Server-Applications
I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ).
A simplified diagram of my ...
- 141
2
votes
0
answers
169
views
Testing an algorithm for safety in mutual exclusion
I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
- 181
1
vote
0
answers
120
views
How does Stackoverflow comment system work
When someone comments on my question/answer/comment I will see the notification from the inbox, and it also shows under my response tab. Later when the comment is edited, the comment shown under the ...
- 111
1
vote
0
answers
27
views
Do Multi-Primary replicated distributed systems guarantee sequential consistency?
I know that Primary-backup replicated distributed systems guarantee sequential consistency.
My question is whether multi-primary systems also achieve that.
I mean if they use consensus (i.e: Paxos ...
1
vote
1
answer
377
views
Regarding data modeling in a NoSQL use case
I am reading the following link to learn about system designs of various systems. (This is a paid link, I am attaching all explanations below.)
In an attempt to explain the system design of Instagram ...
- 27
1
vote
0
answers
50
views
Byzantine Failures - Client Voter and Replica on Same Processor
I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine
Approach: A Tutorial
Dependent-Failures Output ...
- 117
1
vote
1
answer
294
views
Processing data in a cluster with multiple tasks
The goal is to provide a scaleable system implementing multiple data processing tasks which can be seen as a graph. Data objects will travel that graph. Most object will visit the same nodes in the ...
- 237
1
vote
1
answer
1k
views
RabbitMQ: how to achieve scalable one-to-one communication?
I'm trying to use RabbitMQ for two-way one-to-one communication between clients and servers. So, Client 1 should be able to communicate only with Server 1, and vice versa:
The servers stay always ...
- 161
1
vote
0
answers
510
views
In a distributed system, how to prevent duplicated services from acting multiple times on a common data source?
We are currently rewriting our infrastructure from a monolith to a distributed system because the old system didn`t fit the workload anymore.
In one part of our distributed system, we have an Entity (...
- 191
1
vote
0
answers
102
views
Scaling Systems With Shared Data
I'm working on an in memory on disk cache and am starting to think about strategies for scaling the system. At the moment the system persists data by creating a file for each key value pair, where the ...
- 121
0
votes
0
answers
32
views
What is the best way to desgin my api workflow?
I am building a bank management system using the microservices architecture. I have many microservices but I am going to talk about two. I have customer account service and transaction service. Let's ...
- 17
0
votes
0
answers
44
views
Design to handle database race conditions with multiple service instances on multiple clusters
This might be a classic problem in distributed systems but I am sort of lost. We have a kafka consumer in a Java - Spring Boot service that has multiple instances in multiple clusters (all with same ...
- 191
0
votes
0
answers
49
views
Cloud solution to parse and process 1M+ rows
We are writing a service that can parse a CSV file containing over 1M+ rows. Each row needs insertion (or update) data of a DB record. We are currently using DynamoDB.
What AWS services are best ...
- 117
0
votes
1
answer
44
views
How to load consistent test data into multiple applications (which communicate by message streams)?
I have a system landscape consisting of multiple applications.
Each application provides REST interfaces to communicate with the outside world.
Each application has its own data storage (some kind of ...
- 157
0
votes
0
answers
85
views
Bi-directional messaging using Zero MQ pub sub pattern
I am trying to design a system that can be safely used to share messages between processes on the same system using ZeroMQ.
I have chosen to use a central broker that binds to a particular port.
All ...
0
votes
0
answers
66
views
Order notification system in distributed systems
I'm working in a company that has a driver management product. To deliver purchases there is a job in the current system that sends notifications to couriers, stating there is a new purchase to be ...
0
votes
0
answers
111
views
Dealing with third party api rate limit when using multiple services
I have 1 api key that has a rate limit 1000req/1min.
I would like to somehow use 50% of that limit in service 1 and 50% of that limit in service 2. Let's say they're just 2 separate processes using ...
- 1,431
0
votes
0
answers
79
views
Load balancing the load balancers
Currently I have a system where I have installed HAProxy on one machine and my other 3 machines serves the webapps and the fourth machine for the database. Now I need to add another load balancer in ...
- 109
0
votes
1
answer
64
views
Is publishing to the same event store stream from multiple places an anti pattern in a distributed transaction?
Below is a little context to the problem I'm trying to solve.
There are two separate microservices with separate frontends , APIs and databases.
Customer Microservice - Manages customers
Orders ...
- 302
0
votes
1
answer
159
views
How to manage a distributed application?
I have a distributed application backed by a distributed database. (Distributed here means running across multiple hosts with one or more instances on each host). Application has several toolings ...
0
votes
0
answers
52
views
Realtime report generation and downloads
We host a dashboard where users can view different reports, each powered by a templated SQL query. The users have an option to "download" those reports as CSV.
Currently the above use case ...
- 101
0
votes
0
answers
57
views
Experiences with coarse grained and fine grained message designs in an ECST pattern
I am currently facing a dilemma in the design of a message oriented distributed system following an Event Carried State Transfer pattern. This is essentially a "near real time ETL" used to ...
- 306
0
votes
0
answers
472
views
log centralization and central aggregation system design
I'm thinking of how would I go about creating my own logging centralization and centralized aggregation system.
Imagined system has 10 - 100 nodes. Each node runs a LAMP(Linux, Apache, MySql, PHP) ...
- 109
0
votes
0
answers
32
views
Turn based access to update shared resource such as inventory
I am trying to design a system where a Product can only be sold while inventory is available. This is an e-commerce platform where users can buy specific products with designated inventory. A similar ...
- 238
0
votes
0
answers
159
views
What is a good design for maintaining consistency in a last write wins database without event sourcing
We have an application which on every resource update(resource update, create, delete)creates a message and puts it on a queue. This message is picked up by a worker process, converted to a format ...
- 844
0
votes
0
answers
49
views
How should the interaction between the MIS instances of a large system happen?
Overall situations:
The plan was to build a large system (Management Information System or MIS type) to be used throughout the country, by say 100,000 users daily. The team decided to split that into ...
- 119
0
votes
0
answers
41
views
How to arrange synchronicity reporting in an eventually consistent, distributed system?
Context: A typical corporate landscape with many systems located in different data centers, with firewalls, without uniformised interfacing.
Some data X is being replicated for availability and ...
- 101
0
votes
0
answers
42
views
Do web companies use the master replica in data center nearest to the user to do writes?
I am interested how multi datacenter (multi homing in Google terms) systems handle writes to strongly consistent user data? Is the master replica for each user placed in the same datacenter and writes ...
- 1
0
votes
0
answers
113
views
How to implement a fair queue, for multitanent, limitation per machine and limitation cross cluster?
I'm maintaining an already built system. The current system operates as following:
Current situation:
Client (UI) submits a request.
Web layer round-robins to one of backends.
Backend takes request ...
- 507
-1
votes
1
answer
131
views
Checking a related aggregate existence is Domain concern?
This is my specific situation:
I am following DDD and Event sourcing to implement my application (using C#). I have a Domain layer that contains Aggregates' implementation. I write unit tests for this ...
- 99
-1
votes
1
answer
67
views
How to design a topped-up internal balance for a replicated API?
My product is an API which charges an amount of USD every N requests.
I would like to allow my users to top-up their internal balances through PayPal, then decrease their balances when they use the ...
- 99
-1
votes
1
answer
51
views
Whats the pros and cons of RPCs vs. HTTP in information exchange in microservice architectures?
What type of components could benefit from RPC over HTTP, vice-versa? Does there exist other information exchange mechanisms, what are their use?
I've seen smart people use both, I'm sure for ...
- 11