Questions tagged [distributed-system]

This tag can be used by questions on distributed system concept, design, and implementations.

45 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4 votes
1 answer
250 views

Can an api gateway handle websockets requests besides regular http requests at the same time?

I am working on social platform with a distributed architecture. The platform should offer a chat module, but since the most efficient way to implement a chat is using websockets. I am not sure what ...
user avatar
  • 149
3 votes
0 answers
83 views

Difficulty understanding how federation increases cache performance for databases?

I am studying system design for distributed systems and in this page (https://github.com/donnemartin/system-design-primer), one of the following advantages was mentioned for federation for databases ...
user avatar
  • 153
3 votes
0 answers
53 views

Release Phase in Lamport's Mutual Exclusion Algorithm

I was studying Lamport's mutual exclusion algorithm from the original paper, and noticed a difference in the release phase to the one given in Wikipedia. In the original paper, author states that : ...
user avatar
  • 181
3 votes
0 answers
66 views

Sharing tasks among parallel independent jobs

I am getting files on HDFS and I have a legacy java app which checks for new files every 15 mins, does some transformations and write them to some remote server. Now, I am facing scaling issue with ...
user avatar
  • 209
3 votes
0 answers
84 views

Architecture and implementation of a de-centralised web service

I'm a web programmer with several years of experience in RoR and node.js. I have a project idea that is quite big and I need some input in how it can be achieved. I heard of the Diaspora project, and ...
user avatar
  • 291
3 votes
0 answers
714 views

Vector clock conflict resolution in event sourced system via local timestamp - is this safe?

We have an occasionally connected, event sourced system: a budgeting application where each client can pull other client's event store from a separate storage (network share, dropbox,...). It is our ...
user avatar
3 votes
1 answer
151 views

Database consistency for a distributed system

I'm working on a phone call monitoring project. The aim is to have one row in the calls table for each ongoing call. Each call instance may be updated by many different servers/threads as more ...
user avatar
  • 141
2 votes
0 answers
230 views

System Design: Chat application in a distributed system context

I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
user avatar
2 votes
0 answers
90 views

Automatically extended database-based mutex lock - is it a good idea?

Since I am working in a multi-instance microservice environment, I came accross a problem with making some operations being performed at most by one of the running instances at once. The solution ...
user avatar
  • 129
2 votes
0 answers
70 views

SAGA pattern - how to handle situation when some message is not received by orchestrator

Recently I'm studying Saga pattern, but I'm struggling to imagine how it will work in real system. Standard example is making an order, which consist of reserving flight, hotel and a car. We can ...
user avatar
2 votes
1 answer
412 views

What are the different messaging patterns for multiple producers to single consumer through a distributed queue?

We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
user avatar
  • 129
2 votes
0 answers
71 views

Distributing tasks to several clients

I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements. I have 1-...
user avatar
2 votes
1 answer
347 views

Design question on synchronization of two asynchronous data streams

I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
user avatar
2 votes
3 answers
2k views

Message Broker in Client-Server-Applications

I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ). A simplified diagram of my ...
user avatar
  • 141
2 votes
0 answers
169 views

Testing an algorithm for safety in mutual exclusion

I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
user avatar
  • 181
1 vote
0 answers
120 views

How does Stackoverflow comment system work

When someone comments on my question/answer/comment I will see the notification from the inbox, and it also shows under my response tab. Later when the comment is edited, the comment shown under the ...
user avatar
  • 111
1 vote
0 answers
27 views

Do Multi-Primary replicated distributed systems guarantee sequential consistency?

I know that Primary-backup replicated distributed systems guarantee sequential consistency. My question is whether multi-primary systems also achieve that. I mean if they use consensus (i.e: Paxos ...
user avatar
1 vote
1 answer
377 views

Regarding data modeling in a NoSQL use case

I am reading the following link to learn about system designs of various systems. (This is a paid link, I am attaching all explanations below.) In an attempt to explain the system design of Instagram ...
user avatar
1 vote
0 answers
50 views

Byzantine Failures - Client Voter and Replica on Same Processor

I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial Dependent-Failures Output ...
user avatar
  • 117
1 vote
1 answer
294 views

Processing data in a cluster with multiple tasks

The goal is to provide a scaleable system implementing multiple data processing tasks which can be seen as a graph. Data objects will travel that graph. Most object will visit the same nodes in the ...
user avatar
  • 237
1 vote
1 answer
1k views

RabbitMQ: how to achieve scalable one-to-one communication?

I'm trying to use RabbitMQ for two-way one-to-one communication between clients and servers. So, Client 1 should be able to communicate only with Server 1, and vice versa: The servers stay always ...
user avatar
  • 161
1 vote
0 answers
510 views

In a distributed system, how to prevent duplicated services from acting multiple times on a common data source?

We are currently rewriting our infrastructure from a monolith to a distributed system because the old system didn`t fit the workload anymore. In one part of our distributed system, we have an Entity (...
user avatar
  • 191
1 vote
0 answers
102 views

Scaling Systems With Shared Data

I'm working on an in memory on disk cache and am starting to think about strategies for scaling the system. At the moment the system persists data by creating a file for each key value pair, where the ...
user avatar
  • 121
0 votes
0 answers
32 views

What is the best way to desgin my api workflow?

I am building a bank management system using the microservices architecture. I have many microservices but I am going to talk about two. I have customer account service and transaction service. Let's ...
user avatar
0 votes
0 answers
44 views

Design to handle database race conditions with multiple service instances on multiple clusters

This might be a classic problem in distributed systems but I am sort of lost. We have a kafka consumer in a Java - Spring Boot service that has multiple instances in multiple clusters (all with same ...
user avatar
  • 191
0 votes
0 answers
49 views

Cloud solution to parse and process 1M+ rows

We are writing a service that can parse a CSV file containing over 1M+ rows. Each row needs insertion (or update) data of a DB record. We are currently using DynamoDB. What AWS services are best ...
user avatar
0 votes
1 answer
44 views

How to load consistent test data into multiple applications (which communicate by message streams)?

I have a system landscape consisting of multiple applications. Each application provides REST interfaces to communicate with the outside world. Each application has its own data storage (some kind of ...
user avatar
0 votes
0 answers
85 views

Bi-directional messaging using Zero MQ pub sub pattern

I am trying to design a system that can be safely used to share messages between processes on the same system using ZeroMQ. I have chosen to use a central broker that binds to a particular port. All ...
user avatar
0 votes
0 answers
66 views

Order notification system in distributed systems

I'm working in a company that has a driver management product. To deliver purchases there is a job in the current system that sends notifications to couriers, stating there is a new purchase to be ...
user avatar
0 votes
0 answers
111 views

Dealing with third party api rate limit when using multiple services

I have 1 api key that has a rate limit 1000req/1min. I would like to somehow use 50% of that limit in service 1 and 50% of that limit in service 2. Let's say they're just 2 separate processes using ...
user avatar
  • 1,431
0 votes
0 answers
79 views

Load balancing the load balancers

Currently I have a system where I have installed HAProxy on one machine and my other 3 machines serves the webapps and the fourth machine for the database. Now I need to add another load balancer in ...
user avatar
0 votes
1 answer
64 views

Is publishing to the same event store stream from multiple places an anti pattern in a distributed transaction?

Below is a little context to the problem I'm trying to solve. There are two separate microservices with separate frontends , APIs and databases. Customer Microservice - Manages customers Orders ...
user avatar
  • 302
0 votes
1 answer
159 views

How to manage a distributed application?

I have a distributed application backed by a distributed database. (Distributed here means running across multiple hosts with one or more instances on each host). Application has several toolings ...
user avatar
0 votes
0 answers
52 views

Realtime report generation and downloads

We host a dashboard where users can view different reports, each powered by a templated SQL query. The users have an option to "download" those reports as CSV. Currently the above use case ...
user avatar
  • 101
0 votes
0 answers
57 views

Experiences with coarse grained and fine grained message designs in an ECST pattern

I am currently facing a dilemma in the design of a message oriented distributed system following an Event Carried State Transfer pattern. This is essentially a "near real time ETL" used to ...
user avatar
  • 306
0 votes
0 answers
472 views

log centralization and central aggregation system design

I'm thinking of how would I go about creating my own logging centralization and centralized aggregation system. Imagined system has 10 - 100 nodes. Each node runs a LAMP(Linux, Apache, MySql, PHP) ...
user avatar
  • 109
0 votes
0 answers
32 views

Turn based access to update shared resource such as inventory

I am trying to design a system where a Product can only be sold while inventory is available. This is an e-commerce platform where users can buy specific products with designated inventory. A similar ...
user avatar
0 votes
0 answers
159 views

What is a good design for maintaining consistency in a last write wins database without event sourcing

We have an application which on every resource update(resource update, create, delete)creates a message and puts it on a queue. This message is picked up by a worker process, converted to a format ...
user avatar
  • 844
0 votes
0 answers
49 views

How should the interaction between the MIS instances of a large system happen?

Overall situations: The plan was to build a large system (Management Information System or MIS type) to be used throughout the country, by say 100,000 users daily. The team decided to split that into ...
user avatar
0 votes
0 answers
41 views

How to arrange synchronicity reporting in an eventually consistent, distributed system?

Context: A typical corporate landscape with many systems located in different data centers, with firewalls, without uniformised interfacing. Some data X is being replicated for availability and ...
user avatar
0 votes
0 answers
42 views

Do web companies use the master replica in data center nearest to the user to do writes?

I am interested how multi datacenter (multi homing in Google terms) systems handle writes to strongly consistent user data? Is the master replica for each user placed in the same datacenter and writes ...
user avatar
  • 1
0 votes
0 answers
113 views

How to implement a fair queue, for multitanent, limitation per machine and limitation cross cluster?

I'm maintaining an already built system. The current system operates as following: Current situation: Client (UI) submits a request. Web layer round-robins to one of backends. Backend takes request ...
user avatar
  • 507
-1 votes
1 answer
131 views

Checking a related aggregate existence is Domain concern?

This is my specific situation: I am following DDD and Event sourcing to implement my application (using C#). I have a Domain layer that contains Aggregates' implementation. I write unit tests for this ...
user avatar
  • 99
-1 votes
1 answer
67 views

How to design a topped-up internal balance for a replicated API?

My product is an API which charges an amount of USD every N requests. I would like to allow my users to top-up their internal balances through PayPal, then decrease their balances when they use the ...
user avatar
  • 99
-1 votes
1 answer
51 views

Whats the pros and cons of RPCs vs. HTTP in information exchange in microservice architectures?

What type of components could benefit from RPC over HTTP, vice-versa? Does there exist other information exchange mechanisms, what are their use? I've seen smart people use both, I'm sure for ...
user avatar