Questions tagged [distributed-system]

This tag can be used by questions on distributed system concept, design, and implementations.

Filter by
Sorted by
Tagged with
1 vote
1 answer
377 views

Regarding data modeling in a NoSQL use case

I am reading the following link to learn about system designs of various systems. (This is a paid link, I am attaching all explanations below.) In an attempt to explain the system design of Instagram ...
-2 votes
0 answers
79 views

Using a hybrid watermarking/fingerprinting approach for identifying GPS geolocation data [closed]

Our backend is made up of thousands of micro-services. My team deals with and owns geolocation data for our entire backend ecosystem. We get different types of geolocation data from customers: GPS, ...
2 votes
1 answer
347 views

Design question on synchronization of two asynchronous data streams

I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
1 vote
1 answer
1k views

RabbitMQ: how to achieve scalable one-to-one communication?

I'm trying to use RabbitMQ for two-way one-to-one communication between clients and servers. So, Client 1 should be able to communicate only with Server 1, and vice versa: The servers stay always ...
-3 votes
1 answer
232 views

Is this the right mechanism for 2 way heartbeat?

What is the right way to do it for a service runing on 2 places?
2 votes
3 answers
2k views

Message Broker in Client-Server-Applications

I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ). A simplified diagram of my ...
1 vote
1 answer
294 views

Processing data in a cluster with multiple tasks

The goal is to provide a scaleable system implementing multiple data processing tasks which can be seen as a graph. Data objects will travel that graph. Most object will visit the same nodes in the ...
0 votes
1 answer
44 views

How to load consistent test data into multiple applications (which communicate by message streams)?

I have a system landscape consisting of multiple applications. Each application provides REST interfaces to communicate with the outside world. Each application has its own data storage (some kind of ...
4 votes
1 answer
250 views

Can an api gateway handle websockets requests besides regular http requests at the same time?

I am working on social platform with a distributed architecture. The platform should offer a chat module, but since the most efficient way to implement a chat is using websockets. I am not sure what ...
0 votes
1 answer
64 views

Is publishing to the same event store stream from multiple places an anti pattern in a distributed transaction?

Below is a little context to the problem I'm trying to solve. There are two separate microservices with separate frontends , APIs and databases. Customer Microservice - Manages customers Orders ...
0 votes
1 answer
159 views

How to manage a distributed application?

I have a distributed application backed by a distributed database. (Distributed here means running across multiple hosts with one or more instances on each host). Application has several toolings ...
0 votes
0 answers
49 views

Cloud solution to parse and process 1M+ rows

We are writing a service that can parse a CSV file containing over 1M+ rows. Each row needs insertion (or update) data of a DB record. We are currently using DynamoDB. What AWS services are best ...
2 votes
1 answer
411 views

What are the different messaging patterns for multiple producers to single consumer through a distributed queue?

We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
-1 votes
1 answer
77 views

Design approach to decouple DB write to reduce application latency

I want to use DynamoDB to save application operation data at the end of a session for auditing purposes without affecting the actual application latency. Because this date is only for auditing so we ...
3 votes
0 answers
83 views

Difficulty understanding how federation increases cache performance for databases?

I am studying system design for distributed systems and in this page (https://github.com/donnemartin/system-design-primer), one of the following advantages was mentioned for federation for databases ...
0 votes
1 answer
132 views

Consistency and Availability in distributed hashing Key value store

During the last semester of my college, we were asked to read the (chord paper) and implement it without fault tolerance. We built it with the following feature: A new node can join. A client can ...
-1 votes
1 answer
131 views

Checking a related aggregate existence is Domain concern?

This is my specific situation: I am following DDD and Event sourcing to implement my application (using C#). I have a Domain layer that contains Aggregates' implementation. I write unit tests for this ...
-4 votes
3 answers
422 views

How does a distributed system both tolerate network partition and achieve consistency?

In the CAP theorem, there are only three possible cases: C and A without P C and P without A A and P without C How can we have both P and C in the second case? Doesn't propagation of update from ...
0 votes
0 answers
84 views

Bi-directional messaging using Zero MQ pub sub pattern

I am trying to design a system that can be safely used to share messages between processes on the same system using ZeroMQ. I have chosen to use a central broker that binds to a particular port. All ...
0 votes
2 answers
135 views

How is cross device eventual consistency achieved in real time?

Assume there is a distributed system which works on the principles of eventual consistency. Let's consider a scenario where I update my profile picture using my phone. If I log in using another system,...
1 vote
1 answer
117 views

Streaming Market Data Over The Public Internet

I'm currently working on a design for a system that can service market data to external clients over the internet. The current solution involves consuming market data from a feed (Feed Handler), this ...
0 votes
0 answers
66 views

Order notification system in distributed systems

I'm working in a company that has a driver management product. To deliver purchases there is a job in the current system that sends notifications to couriers, stating there is a new purchase to be ...
0 votes
1 answer
119 views

Maintaining Objects Across API Deployment Instances

I am working on a web application as a hobby and trying to learn some concepts related to cloud development and distributed applications. I am currently targeting an AWS EC2 instance as a deployment ...
0 votes
1 answer
53 views

designing an order management and dispatch management system for couriers

I'm studying to develop an order management system and order dispatch for the couriers, this is a personal project, for study purposes. Some important points about the deliverymen, they are bikers and ...
3 votes
5 answers
1k views

How to resolve cyclic dependencies in Event-driven systems?

Consider two services (bounded contexts by DDD): Sales Billing Sales is responsible for creating orders and Billing for handling payments. Sales tracks orders and Billing holds payments: Sales ...
0 votes
0 answers
111 views

Dealing with third party api rate limit when using multiple services

I have 1 api key that has a rate limit 1000req/1min. I would like to somehow use 50% of that limit in service 1 and 50% of that limit in service 2. Let's say they're just 2 separate processes using ...
0 votes
0 answers
79 views

Load balancing the load balancers

Currently I have a system where I have installed HAProxy on one machine and my other 3 machines serves the webapps and the fourth machine for the database. Now I need to add another load balancer in ...
2 votes
0 answers
229 views

System Design: Chat application in a distributed system context

I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
13 votes
2 answers
583 views

When bounded contexts and "microservices" collide. A distributed systems dilemma in diagram form

You can't please everyone. Some people want a lot of context and background on sites like this. Others do not. If you don't want the background, skip the first three paragraphs. I am a software ...
-4 votes
5 answers
771 views

Why can't we vertically scale a system infinitely?

When talking about scaling a system, it's often said that vertical scaling has limitations. So after a point, we need to scale the system horizontally. What are the limitations of: Main memory - ...
-1 votes
1 answer
67 views

How to design a topped-up internal balance for a replicated API?

My product is an API which charges an amount of USD every N requests. I would like to allow my users to top-up their internal balances through PayPal, then decrease their balances when they use the ...
2 votes
2 answers
118 views

Is sequential consistency equivalent to performing memory accesses by a processes in program order and performing each memory access atomically?

In Fundamentals of Parallel Multicore Architecture, by Yan Solihin, p304 defines sequential consistency memory model: Overall, we can express programmers’ implicit expectation of memory access ...
4 votes
2 answers
112 views

Do coherence and transactional isolation mean the same?

About coherence vs consistency, https://en.wikipedia.org/wiki/Consistency_model says Coherence deals with maintaining a global order in which writes to a single location or single variable are seen ...
3 votes
1 answer
560 views

Best strategy to flush Redis cache used in a distributed environment

Current setup: We follow the cache-first strategy right now. Always fetch from the cache first. If absent in cache, fetch from DB and update the cache. Cache TTL is 30 min. After this, the cache-key ...
3 votes
1 answer
175 views

Should user and service-to-service authentications be separate?

Say I have a system with 5 microservices behind a gateway, and a user signs in through an IDP (OAuth) A user U passes the access token in a request, and the call first reaches the gateway before it ...
2 votes
0 answers
90 views

Automatically extended database-based mutex lock - is it a good idea?

Since I am working in a multi-instance microservice environment, I came accross a problem with making some operations being performed at most by one of the running instances at once. The solution ...
0 votes
0 answers
52 views

Realtime report generation and downloads

We host a dashboard where users can view different reports, each powered by a templated SQL query. The users have an option to "download" those reports as CSV. Currently the above use case ...
3 votes
3 answers
785 views

Is "Transactional outbox" an anti-pattern?

"Transactional outbox" is a pattern for distributed systems which makes use of a single db transaction to do both, persist changes to the domain model and send a domain event message to the ...
0 votes
1 answer
197 views

How can message queues improve scalability?

I have read in a DDD book that using message queues between communicating services can make the whole architecture more scalable, amazon's documentation mentions that queues provide granular ...
1 vote
1 answer
472 views

What performance benefits does a Redis cluster have over having all machines connect to a single Redis node? [closed]

Suppose I have data that easily fits in memory in a redis database. The redis server runs as a single service on a single process on a single machine. Let's say I have a 100 different machines connect ...
28 votes
5 answers
6k views

I'm not seeing 'tightly coupled code' as one of the drawbacks of a monolithic application architecture

One of the most common things I see when discussing pros/cons of microservice vs monolithic architecture is that monolithic applications have, or always trend toward, 'tight coupling.' To be honest, I'...
3 votes
1 answer
167 views

How to protect an open-source distributed application consisting of clients and servers from forks made by hackers?

TL;DR: How distributed open-source apps like Scuttlebutt are secured from DoS and hackers who can make custom version of application? I'm struggle with designing an open-source distributed application ...
1 vote
3 answers
169 views

Which code to copy-paste and which to centralize?

In a fairly complex system with a dozen or more git repositories, most of them are done in a cookie-cutter fashion. What is or would be your best approach to deal with duplication? In my opinion the ...
3 votes
1 answer
65 views

How to handle changes to local and global representation of an Entity in a distributed environment?

I'm a developer for a retail enterprise IT department, and am currently stuck on the following scenario: Department A is responsible for a CRUD service that's used for handling stores - so when our ...
3 votes
1 answer
151 views

Database consistency for a distributed system

I'm working on a phone call monitoring project. The aim is to have one row in the calls table for each ongoing call. Each call instance may be updated by many different servers/threads as more ...
7 votes
1 answer
137 views

Architecting basket in high scale distributed application

I'm designing an e-commerce application. Main flow is pretty straightforward: customer add items to basket, checkouts the basket (place an order) and waits for delivery. There are following ...
2 votes
2 answers
368 views

How to discover discovery service

We are designing an architecture of a high availability distibuted system and we are considering server side vs client side service discovery. There is one problem, I need to solve - how is it ...
0 votes
2 answers
484 views

Why is it considered hard to maintain strong consistency in a distributed system?

Why can't you just use strongly consistent reads for all your DB reads, with retries on 500 responses? According to CAP theorem increasing consistency should probably lower availability, but can't the ...
2 votes
1 answer
113 views

Can passive and active replication handle byzantine failures?

Currently studying about replication and I was wondering: In Passive replication, we have an agreement stage where the Primary replica manager (RM) waits for ack from the backup RM, wouldn't it be ...
1 vote
0 answers
120 views

How does Stackoverflow comment system work

When someone comments on my question/answer/comment I will see the notification from the inbox, and it also shows under my response tab. Later when the comment is edited, the comment shown under the ...