Questions tagged [distributed-system]

This tag can be used by questions on distributed system concept, design, and implementations.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Design to prevent unexpected database transactions with multiple service instances on multiple clusters

This might be a classic problem in distributed systems but I am sort of lost. We have a kafka consumer in a Java - Spring Boot service that has multiple instances in multiple clusters (all with same ...
user avatar
  • 183
-2 votes
0 answers
79 views

Using a hybrid watermarking/fingerprinting approach for identifying GPS geolocation data [closed]

Our backend is made up of thousands of micro-services. My team deals with and owns geolocation data for our entire backend ecosystem. We get different types of geolocation data from customers: GPS, ...
user avatar
-3 votes
1 answer
232 views

Is this the right mechanism for 2 way heartbeat?

What is the right way to do it for a service runing on 2 places?
user avatar
0 votes
0 answers
49 views

Cloud solution to parse and process 1M+ rows

We are writing a service that can parse a CSV file containing over 1M+ rows. Each row needs insertion (or update) data of a DB record. We are currently using DynamoDB. What AWS services are best ...
user avatar
0 votes
1 answer
44 views

How to load consistent test data into multiple applications (which communicate by message streams)?

I have a system landscape consisting of multiple applications. Each application provides REST interfaces to communicate with the outside world. Each application has its own data storage (some kind of ...
user avatar
3 votes
0 answers
83 views

Difficulty understanding how federation increases cache performance for databases?

I am studying system design for distributed systems and in this page (https://github.com/donnemartin/system-design-primer), one of the following advantages was mentioned for federation for databases ...
user avatar
  • 153
-1 votes
1 answer
131 views

Checking a related aggregate existence is Domain concern?

This is my specific situation: I am following DDD and Event sourcing to implement my application (using C#). I have a Domain layer that contains Aggregates' implementation. I write unit tests for this ...
user avatar
  • 99
0 votes
0 answers
84 views

Bi-directional messaging using Zero MQ pub sub pattern

I am trying to design a system that can be safely used to share messages between processes on the same system using ZeroMQ. I have chosen to use a central broker that binds to a particular port. All ...
user avatar
0 votes
2 answers
135 views

How is cross device eventual consistency achieved in real time?

Assume there is a distributed system which works on the principles of eventual consistency. Let's consider a scenario where I update my profile picture using my phone. If I log in using another system,...
user avatar
  • 111
1 vote
1 answer
117 views

Streaming Market Data Over The Public Internet

I'm currently working on a design for a system that can service market data to external clients over the internet. The current solution involves consuming market data from a feed (Feed Handler), this ...
user avatar
  • 119
0 votes
0 answers
66 views

Order notification system in distributed systems

I'm working in a company that has a driver management product. To deliver purchases there is a job in the current system that sends notifications to couriers, stating there is a new purchase to be ...
user avatar
0 votes
1 answer
53 views

designing an order management and dispatch management system for couriers

I'm studying to develop an order management system and order dispatch for the couriers, this is a personal project, for study purposes. Some important points about the deliverymen, they are bikers and ...
user avatar
0 votes
0 answers
111 views

Dealing with third party api rate limit when using multiple services

I have 1 api key that has a rate limit 1000req/1min. I would like to somehow use 50% of that limit in service 1 and 50% of that limit in service 2. Let's say they're just 2 separate processes using ...
user avatar
  • 1,431
-1 votes
1 answer
77 views

Design approach to decouple DB write to reduce application latency

I want to use DynamoDB to save application operation data at the end of a session for auditing purposes without affecting the actual application latency. Because this date is only for auditing so we ...
user avatar
0 votes
0 answers
79 views

Load balancing the load balancers

Currently I have a system where I have installed HAProxy on one machine and my other 3 machines serves the webapps and the fourth machine for the database. Now I need to add another load balancer in ...
user avatar
2 votes
0 answers
229 views

System Design: Chat application in a distributed system context

I am currently working on a side project so I can learn new technologies and understand how the components fit together. For this project, I decided that I wanted to work on Chat Applications in the ...
user avatar
0 votes
1 answer
64 views

Is publishing to the same event store stream from multiple places an anti pattern in a distributed transaction?

Below is a little context to the problem I'm trying to solve. There are two separate microservices with separate frontends , APIs and databases. Customer Microservice - Manages customers Orders ...
user avatar
  • 302
4 votes
1 answer
250 views

Can an api gateway handle websockets requests besides regular http requests at the same time?

I am working on social platform with a distributed architecture. The platform should offer a chat module, but since the most efficient way to implement a chat is using websockets. I am not sure what ...
user avatar
  • 149
0 votes
1 answer
159 views

How to manage a distributed application?

I have a distributed application backed by a distributed database. (Distributed here means running across multiple hosts with one or more instances on each host). Application has several toolings ...
user avatar
3 votes
1 answer
560 views

Best strategy to flush Redis cache used in a distributed environment

Current setup: We follow the cache-first strategy right now. Always fetch from the cache first. If absent in cache, fetch from DB and update the cache. Cache TTL is 30 min. After this, the cache-key ...
user avatar
3 votes
1 answer
175 views

Should user and service-to-service authentications be separate?

Say I have a system with 5 microservices behind a gateway, and a user signs in through an IDP (OAuth) A user U passes the access token in a request, and the call first reaches the gateway before it ...
user avatar
2 votes
0 answers
90 views

Automatically extended database-based mutex lock - is it a good idea?

Since I am working in a multi-instance microservice environment, I came accross a problem with making some operations being performed at most by one of the running instances at once. The solution ...
user avatar
  • 129
0 votes
0 answers
52 views

Realtime report generation and downloads

We host a dashboard where users can view different reports, each powered by a templated SQL query. The users have an option to "download" those reports as CSV. Currently the above use case ...
user avatar
  • 101
0 votes
1 answer
197 views

How can message queues improve scalability?

I have read in a DDD book that using message queues between communicating services can make the whole architecture more scalable, amazon's documentation mentions that queues provide granular ...
user avatar
  • 145
1 vote
1 answer
472 views

What performance benefits does a Redis cluster have over having all machines connect to a single Redis node? [closed]

Suppose I have data that easily fits in memory in a redis database. The redis server runs as a single service on a single process on a single machine. Let's say I have a 100 different machines connect ...
user avatar
3 votes
3 answers
785 views

Is "Transactional outbox" an anti-pattern?

"Transactional outbox" is a pattern for distributed systems which makes use of a single db transaction to do both, persist changes to the domain model and send a domain event message to the ...
user avatar
  • 523
1 vote
3 answers
169 views

Which code to copy-paste and which to centralize?

In a fairly complex system with a dozen or more git repositories, most of them are done in a cookie-cutter fashion. What is or would be your best approach to deal with duplication? In my opinion the ...
user avatar
  • 149
2 votes
2 answers
118 views

Is sequential consistency equivalent to performing memory accesses by a processes in program order and performing each memory access atomically?

In Fundamentals of Parallel Multicore Architecture, by Yan Solihin, p304 defines sequential consistency memory model: Overall, we can express programmers’ implicit expectation of memory access ...
user avatar
  • 5,301
4 votes
2 answers
112 views

Do coherence and transactional isolation mean the same?

About coherence vs consistency, https://en.wikipedia.org/wiki/Consistency_model says Coherence deals with maintaining a global order in which writes to a single location or single variable are seen ...
user avatar
  • 5,301
3 votes
1 answer
65 views

How to handle changes to local and global representation of an Entity in a distributed environment?

I'm a developer for a retail enterprise IT department, and am currently stuck on the following scenario: Department A is responsible for a CRUD service that's used for handling stores - so when our ...
user avatar
  • 99
3 votes
1 answer
151 views

Database consistency for a distributed system

I'm working on a phone call monitoring project. The aim is to have one row in the calls table for each ongoing call. Each call instance may be updated by many different servers/threads as more ...
user avatar
  • 141
28 votes
5 answers
6k views

I'm not seeing 'tightly coupled code' as one of the drawbacks of a monolithic application architecture

One of the most common things I see when discussing pros/cons of microservice vs monolithic architecture is that monolithic applications have, or always trend toward, 'tight coupling.' To be honest, I'...
user avatar
  • 421
0 votes
2 answers
484 views

Why is it considered hard to maintain strong consistency in a distributed system?

Why can't you just use strongly consistent reads for all your DB reads, with retries on 500 responses? According to CAP theorem increasing consistency should probably lower availability, but can't the ...
user avatar
  • 421
7 votes
1 answer
137 views

Architecting basket in high scale distributed application

I'm designing an e-commerce application. Main flow is pretty straightforward: customer add items to basket, checkouts the basket (place an order) and waits for delivery. There are following ...
user avatar
2 votes
1 answer
113 views

Can passive and active replication handle byzantine failures?

Currently studying about replication and I was wondering: In Passive replication, we have an agreement stage where the Primary replica manager (RM) waits for ack from the backup RM, wouldn't it be ...
user avatar
  • 131
1 vote
0 answers
120 views

How does Stackoverflow comment system work

When someone comments on my question/answer/comment I will see the notification from the inbox, and it also shows under my response tab. Later when the comment is edited, the comment shown under the ...
user avatar
  • 111
10 votes
2 answers
8k views

What is the meaning of fan-out

I see this word in many places but don't get it. From the Wikipedia, In message-oriented middleware solutions, fan-out is a messaging pattern used to model an information exchange that implies the ...
user avatar
3 votes
1 answer
167 views

How to protect an open-source distributed application consisting of clients and servers from forks made by hackers?

TL;DR: How distributed open-source apps like Scuttlebutt are secured from DoS and hackers who can make custom version of application? I'm struggle with designing an open-source distributed application ...
user avatar
1 vote
1 answer
89 views

Should I use ROA or SOA for a distributed application and how could I implement REST If using ROA

For my final year project I'm looking to build a distributed version of a popular benchmarking client (this has already been done using various methods involving some form of existing frameworks), I ...
user avatar
4 votes
2 answers
99 views

Designing an ETL with where there are a few points of entry

I'm trying to think of a scalable solution for my current system. The current system is 3 microscopes 1 processing machine 1. 60-100GB Files come from 2-3 microscopes every 30 minutes 2. That data ...
user avatar
6 votes
3 answers
290 views

How can adding redunancy adversely affect performance

I have been reading about system design and was going through this popular article - A Word on Scalability The author here says: An always-on service is said to be scalable if adding resources to ...
user avatar
  • 171
2 votes
0 answers
70 views

SAGA pattern - how to handle situation when some message is not received by orchestrator

Recently I'm studying Saga pattern, but I'm struggling to imagine how it will work in real system. Standard example is making an order, which consist of reserving flight, hotel and a car. We can ...
user avatar
1 vote
1 answer
227 views

Distributed message queue, propagating queue leader/follower information

I'm designing a distributed queue message queue and I'm not sure how to get the leader information to another component. This is my current design: I have the following components: FrontEnd: request ...
user avatar
  • 249
0 votes
1 answer
202 views

What should we consider when designing a load balancer with minimum cost

Given a set of processes running on a cluster of hosts, I'm designing a system that load balances the hosts through live migrations (carried out by an external service) of the processes. The goal of ...
user avatar
  • 111
0 votes
1 answer
51 views

Distributed processing of a large number of queues

I have a rather basic application hosted on Kubernetes, which connects to a Mongo database. The app has a wallet feature. A user can put money in their wallet using real-world payments (e.g. via ...
user avatar
  • 357
2 votes
1 answer
91 views

Distributed cart state in microservices

We have legacy app which keeps state of a cart in local storage on a Frontend (this decision was made long time ago by other devs). The cart is cleared from local storage after 2 hours if it is not ...
user avatar
2 votes
1 answer
415 views

Managing consistency in distributed database

I have question regarding system design. Lets say I want to design a Flight booking system ( such as American Airlines). So, consistency when booking a ticket is very important. This booking system is ...
user avatar
0 votes
1 answer
49 views

System design to store pieces of data with specific ttyl to then refetch

I have a system which generates pieces of data that needs to be tracked and assigned a specific TTYL value (a value in seconds) for how long they are supposed to live before being re-processed. I ...
user avatar
-2 votes
2 answers
194 views

How can one choose between consistency and availability if the system you are designing is Tiny URL

I am putting together a system design for something similar to Tiny URL. It provides an API for mapping a URL to a short URL. Once created the short URL can be used to access the original URL. This ...
user avatar
1 vote
1 answer
107 views

Implemeting a distributed n-Producers/1-Consumer service for critical-mission system

I am trying to implement a distributed version of multi-producer/1-consumer for a critical-mission system. I'm looking for good alternatives to the current approach based on RDBMS. The problem The ...
user avatar
  • 113