Questions tagged [distributed-system]

This tag can be used by questions on distributed system concept, design, and implementations.

Filter by
Sorted by
Tagged with
28 votes
5 answers
6k views

I'm not seeing 'tightly coupled code' as one of the drawbacks of a monolithic application architecture

One of the most common things I see when discussing pros/cons of microservice vs monolithic architecture is that monolithic applications have, or always trend toward, 'tight coupling.' To be honest, I'...
Bob Dole's user avatar
  • 421
22 votes
2 answers
17k views

What is the meaning of fan-out

I see this word in many places but don't get it. From the Wikipedia, In message-oriented middleware solutions, fan-out is a messaging pattern used to model an information exchange that implies the ...
user3153970's user avatar
14 votes
5 answers
7k views

How can I efficiently diff a CSV file against a database?

I have an inventory of products stored in Postgres. I need to be able to take a CSV file and get a list of changes—the things in the CSV file that are different to what is in the database. The CSV ...
Isvara's user avatar
  • 630
14 votes
2 answers
857 views

When bounded contexts and "microservices" collide. A distributed systems dilemma in diagram form

You can't please everyone. Some people want a lot of context and background on sites like this. Others do not. If you don't want the background, skip the first three paragraphs. I am a software ...
allmhuran's user avatar
  • 316
13 votes
2 answers
441 views

Does Optimistic Concurrency per object imply Serializability if a transaction will never span multiple objects?

Given a system which provides: Optimistic concurrency control / versioning per object (using CAS - Check-and-Set) Transactions that never need to span more then a single object. Snapshot Isolation ...
Geert-Jan's user avatar
  • 673
12 votes
2 answers
8k views

How to safely run database migrations with multiple app instances?

We have an application that has a mix of both fast (< 1 second) and slow database migrations (> 30 seconds). Right now, we're running database migrations as a part of CI, but then our CI tool has ...
Benjewman's user avatar
  • 313
10 votes
3 answers
2k views

How do distributed databases follow unique constraints?

Lets say i have an application where user can register, and the username has to be unqiue value. Now lets say i have N partitions and for each partition i have M replicas with multiple leaders. Now i ...
Johnyb's user avatar
  • 407
10 votes
3 answers
8k views

How acceptable is to keep business logic outside entities (in separate service classes)?

We were taught that objects are self contained things with data and behaviour and therefore they should have methods that act on their attributes. But there are several situations when this coupling ...
TomR's user avatar
  • 1,003
8 votes
4 answers
1k views

If a system talks to a database to get some previous information to serve a request, does that make the system **stateful** or **stateless**?

I've read this example which basically states that, since a method has to hold a single private field.. It is considered a stateful system. So basically ANYTHING system that calls upon some sort of ...
imperialgendarme's user avatar
7 votes
3 answers
9k views

Best practices for Heartbeat in distributed systems

We had in our system in the past an external data provider (call it source) sending regular heartbeats to a java application (call it client). If the heartbeat failed, system shut itself down (to ...
senseiwu's user avatar
  • 668
7 votes
2 answers
4k views

How do you solve issue of consistency in concurrent and distributed application (built around Bankers Dilemma)?

This is a classic problem which I'm sure has been solved many times by many different people. I don't have any formal training (I've not studied computer science or any other such academic subject) ...
Integralist's user avatar
7 votes
1 answer
242 views

Architecting basket in high scale distributed application

I'm designing an e-commerce application. Main flow is pretty straightforward: customer add items to basket, checkouts the basket (place an order) and waits for delivery. There are following ...
Maciej Pszczolinski's user avatar
7 votes
3 answers
3k views

How to prevent overlap booking on a calendar booking system

As an exercise, I am trying to design a simple calendar booking system for multiple meeting rooms. I kind of got my head around some requirements such as find available rooms for a given time range, ...
smurf's user avatar
  • 79
6 votes
3 answers
305 views

How can adding redunancy adversely affect performance

I have been reading about system design and was going through this popular article - A Word on Scalability The author here says: An always-on service is said to be scalable if adding resources to ...
sarincasm's user avatar
  • 171
6 votes
1 answer
939 views

Should user and service-to-service authentications be separate?

Say I have a system with 5 microservices behind a gateway, and a user signs in through an IDP (OAuth) A user U passes the access token in a request, and the call first reaches the gateway before it ...
Jerald Baker's user avatar
6 votes
1 answer
969 views

Autosuggest at scale - trie sharding

While reading on the design for autosuggest implementation on large scale systems (like google), I'm able to understand the usage of trie and how top "n" terms are stored at each node to quickly ...
user2599672's user avatar
5 votes
3 answers
845 views

How does a CDN get "data" when there's hundreds of terabytes stored in data centers?

I am still trying to wrap my head around CDN's in a real world scenario. Suppose I am building a Netflix clone. I have about 1,000 terabytes of video content stored in an S3 bucket. Since Netflix ...
dsomel21's user avatar
  • 171
5 votes
2 answers
3k views

Alternatives to distributed transactions in .NET?

Lately, I've been working on a project which basically is a huge rewrite in .NET Core F# + Event Sourcing + PostgreSQL of an old sub-ledger legacy app written in .NET 4.6 C# + SQL Server. Since the ...
Natalie Perret's user avatar
5 votes
1 answer
599 views

Can an api gateway handle websockets requests besides regular http requests at the same time?

I am working on social platform with a distributed architecture. The platform should offer a chat module, but since the most efficient way to implement a chat is using websockets. I am not sure what ...
cristid9's user avatar
  • 159
5 votes
1 answer
2k views

How does Redis (or any typical distributed cache) handle replication conflicts?

Suppose you set up a Redis cluster with one master and two slaves. Two clients are connected to each of the slaves. Both clients make conflicting changes at the same time: What happens if these ...
Joey Adams's user avatar
  • 5,585
5 votes
1 answer
202 views

Is the actor model suited for an application dealing with elapsed time?

I need to create a distributed application with timers / periodically checking two timestamps for elapsed time. Is it possible to achieve following behaviour with the actor model: Spawn Child actor ...
Towen's user avatar
  • 59
4 votes
3 answers
387 views

How to think about system design questions methodically?

I'm working on my first big system design project at work for the first time in my career. In the past, while i've worked on large systems, I was always working with someone else, so I wasn't really ...
Jeremy Fisher's user avatar
4 votes
3 answers
242 views

Consistency of diamonds in a concurrent system

I am struggling to find solutions to a problem that I encounter in a concurrent system (I am currently using the Actor Model). I basically have a diamond structure of actors: So D sends messages to C ...
Marco's user avatar
  • 149
4 votes
1 answer
759 views

Managing consistency in distributed database

I have question regarding system design. Lets say I want to design a Flight booking system ( such as American Airlines). So, consistency when booking a ticket is very important. This booking system is ...
StackManaged's user avatar
4 votes
2 answers
4k views

Architecture for message processing with scheduling, at scale

I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with ...
rubik's user avatar
  • 143
4 votes
2 answers
745 views

How to implement to flood fill algorithm on multiple machines?

I have a large image distributed over multiple machines for which I need to implement the flood fill algorithm used in MS Paint. I am able to do it with a single machine but what approach must be ...
Nikant's user avatar
  • 139
4 votes
1 answer
157 views

How are application in separate data centers routed

A bit of down time at work let my mind wonder a bit. I was designing a single location application on Digital Ocean (load balancer to app servers to DB with DNS). All fine. Then I thought what if the ...
nerdlyist's user avatar
  • 945
4 votes
2 answers
113 views

Designing an ETL with where there are a few points of entry

I'm trying to think of a scalable solution for my current system. The current system is 3 microscopes 1 processing machine 1. 60-100GB Files come from 2-3 microscopes every 30 minutes 2. That data ...
user3145912's user avatar
4 votes
1 answer
851 views

distributed application design - using message broker

Background I'm trying to design a layer/component in my distributed application that will communicate between component A and component B. Right now, this "communication" is accomplished by ...
dot's user avatar
  • 561
4 votes
2 answers
145 views

Do coherence and transactional isolation mean the same?

About coherence vs consistency, https://en.wikipedia.org/wiki/Consistency_model says Coherence deals with maintaining a global order in which writes to a single location or single variable are seen ...
Tim's user avatar
  • 5,465
4 votes
3 answers
432 views

Distributed database algorithm

I have a need for a distributed data store, where existing solutions may not work as the computers these will be running on will be extremely resource limited, for instance 64-128MB RAM. Plus, as a ...
C B's user avatar
  • 149
4 votes
2 answers
108 views

Performance considerations with access control in a distributed system architecture

We're currently working on upgrading a small information system for an enterprise. Currently, the system has a corporate network zone (CORP) where services for internal users are placed, and a data ...
NLuburić's user avatar
  • 191
4 votes
2 answers
217 views

How do replicas of an app instance assure the task was done only once when the main instance fails?

Let's assume i app have with basic flow: Client send request to server Server do some business logic Server puts message into some message queue Server returns success to the client microservice A ...
Johnyb's user avatar
  • 407
3 votes
3 answers
2k views

Is "Transactional outbox" an anti-pattern?

"Transactional outbox" is a pattern for distributed systems which makes use of a single db transaction to do both, persist changes to the domain model and send a domain event message to the ...
nix's user avatar
  • 533
3 votes
5 answers
2k views

How to resolve cyclic dependencies in Event-driven systems?

Consider two services (bounded contexts by DDD): Sales Billing Sales is responsible for creating orders and Billing for handling payments. Sales tracks orders and Billing holds payments: Sales ...
Barney's user avatar
  • 211
3 votes
2 answers
197 views

How to propagate seldomly changing data in a distributed system

We store data – some kind of metadata like car manufacturers or types of computer parts – in one of our applications. This data changes rarely. Let's say it changes once every two weeks. This data is ...
Harold L. Brown's user avatar
3 votes
2 answers
1k views

Decrement counter with high concurrency in distributed system

I am working on problem where a call to reduce a counter will come to a service and if counter is greater zero then call should be able to reduce it else fail. Pretty straightforward? huh! For a ...
gitesh.tyagi's user avatar
3 votes
2 answers
979 views

Thinking pragmatically about scalability

I was recently asked as an exercise to design a scalable graph. My first intuition was how to seperate the graph and distribute it (sharding,consistent hashing..etc) Turns out my thinking was on ...
Stan R.'s user avatar
  • 133
3 votes
2 answers
2k views

Store file in filesystem, and its metadata to the database atomicly

I have to store many pdf/jpg/png file of max 10mb in a filesystem, and need to save their metadata on a database. The SFTP and the DB may be on different nodes. On WS, I've a local db where I can ...
Federico Ponzi's user avatar
3 votes
1 answer
203 views

The publish-subscribe pattern: garbage-collecting old subscriptions

I've been studying distributed systems design, in particular the Udi Dahan's class. He talks about the publish-subscribe pattern as a common pattern in messaging-oriented designs. There's obviously a ...
liori's user avatar
  • 725
3 votes
2 answers
170 views

load balancing in shared nothing cluster

I have a system where the load is distributed across the nodes in the cluster and once an entity is assigned to particular node, it is taken to completion through that node only(distribution is saved ...
user2599052's user avatar
3 votes
1 answer
2k views

Best strategy to flush Redis cache used in a distributed environment

Current setup: We follow the cache-first strategy right now. Always fetch from the cache first. If absent in cache, fetch from DB and update the cache. Cache TTL is 30 min. After this, the cache-key ...
YetAnotherBot's user avatar
3 votes
1 answer
67 views

How to handle changes to local and global representation of an Entity in a distributed environment?

I'm a developer for a retail enterprise IT department, and am currently stuck on the following scenario: Department A is responsible for a CRUD service that's used for handling stores - so when our ...
sigsegv's user avatar
  • 99
3 votes
1 answer
180 views

How to protect an open-source distributed application consisting of clients and servers from forks made by hackers?

TL;DR: How distributed open-source apps like Scuttlebutt are secured from DoS and hackers who can make custom version of application? I'm struggle with designing an open-source distributed application ...
Daniil Miroshnykov's user avatar
3 votes
2 answers
572 views

Synchronous communcation vs caching in microservices

I'm trying to build a distributed system and have been researching how best to manage dependencies between services. The common example in tutorials is an ordering system - let's say I have a Catalog, ...
Laurence's user avatar
  • 133
3 votes
2 answers
735 views

Architecting a distributed file processing system with leadership election

I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice: Problem : There are a large number of files continuously posted on ...
user8811409's user avatar
3 votes
2 answers
612 views

What are the ideal use cases for weakly consistent systems?

I have been reading about distributed databases and the like providing weak and strong consistency guarantees, but perhaps I am short on imagination - when would you want to use a weakly consistent ...
Akshat Mahajan's user avatar
3 votes
1 answer
290 views

Is TCP still considered siginificant overhead compared to UDP? [closed]

I'm working on a distributed application for a graduate class I'm taking. There has been much discussion around implementing a way to maintain peer connection health statuses. Think Gossip. One of the ...
ctote's user avatar
  • 173
3 votes
3 answers
2k views

Multiple Databases per Microservice

We have a scenario in which all the important and transactional fields of our business entities are highly structured and relational. The data size of these important fields is also very small. ...
iavanish's user avatar
  • 139
3 votes
1 answer
1k views

Configuration in a Distributed Application

I am developing a distributed application built as a collection of separate services which are served from multiple load balanced instances and trying to wrap my head around the distributed ...
Jonathan Gross's user avatar