Questions tagged [distributed-system]

This tag can be used by questions on distributed system concept, design, and implementations.

Filter by
Sorted by
Tagged with
28 votes
5 answers
6k views

I'm not seeing 'tightly coupled code' as one of the drawbacks of a monolithic application architecture

One of the most common things I see when discussing pros/cons of microservice vs monolithic architecture is that monolithic applications have, or always trend toward, 'tight coupling.' To be honest, I'...
user avatar
  • 421
13 votes
2 answers
583 views

When bounded contexts and "microservices" collide. A distributed systems dilemma in diagram form

You can't please everyone. Some people want a lot of context and background on sites like this. Others do not. If you don't want the background, skip the first three paragraphs. I am a software ...
user avatar
  • 306
13 votes
2 answers
397 views

Does Optimistic Concurrency per object imply Serializability if a transaction will never span multiple objects?

Given a system which provides: Optimistic concurrency control / versioning per object (using CAS - Check-and-Set) Transactions that never need to span more then a single object. Snapshot Isolation ...
user avatar
  • 663
12 votes
2 answers
7k views

How to safely run database migrations with multiple app instances?

We have an application that has a mix of both fast (< 1 second) and slow database migrations (> 30 seconds). Right now, we're running database migrations as a part of CI, but then our CI tool has ...
user avatar
  • 313
10 votes
2 answers
8k views

What is the meaning of fan-out

I see this word in many places but don't get it. From the Wikipedia, In message-oriented middleware solutions, fan-out is a messaging pattern used to model an information exchange that implies the ...
user avatar
10 votes
3 answers
7k views

How acceptable is to keep business logic outside entities (in separate service classes)?

We were taught that objects are self contained things with data and behaviour and therefore they should have methods that act on their attributes. But there are several situations when this coupling ...
user avatar
  • 1,009
7 votes
1 answer
137 views

Architecting basket in high scale distributed application

I'm designing an e-commerce application. Main flow is pretty straightforward: customer add items to basket, checkouts the basket (place an order) and waits for delivery. There are following ...
user avatar
7 votes
2 answers
2k views

How to prevent overlap booking on a calendar booking system

As an exercise, I am trying to design a simple calendar booking system for multiple meeting rooms. I kind of got my head around some requirements such as find available rooms for a given time range, ...
user avatar
  • 79
6 votes
3 answers
290 views

How can adding redunancy adversely affect performance

I have been reading about system design and was going through this popular article - A Word on Scalability The author here says: An always-on service is said to be scalable if adding resources to ...
user avatar
  • 171
6 votes
2 answers
3k views

How do you solve issue of consistency in concurrent and distributed application (built around Bankers Dilemma)?

This is a classic problem which I'm sure has been solved many times by many different people. I don't have any formal training (I've not studied computer science or any other such academic subject) ...
user avatar
6 votes
1 answer
763 views

Autosuggest at scale - trie sharding

While reading on the design for autosuggest implementation on large scale systems (like google), I'm able to understand the usage of trie and how top "n" terms are stored at each node to quickly ...
user avatar
5 votes
3 answers
6k views

Best practices for Heartbeat in distributed systems

We had in our system in the past an external data provider (call it source) sending regular heartbeats to a java application (call it client). If the heartbeat failed, system shut itself down (to ...
user avatar
  • 638
4 votes
3 answers
430 views

How does a CDN get "data" when there's hundreds of terabytes stored in data centers?

I am still trying to wrap my head around CDN's in a real world scenario. Suppose I am building a Netflix clone. I have about 1,000 terabytes of video content stored in an S3 bucket. Since Netflix ...
user avatar
  • 161
4 votes
4 answers
623 views

If a system talks to a database to get some previous information to serve a request, does that make the system **stateful** or **stateless**?

I've read this example which basically states that, since a method has to hold a single private field.. It is considered a stateful system. So basically ANYTHING system that calls upon some sort of ...
user avatar
4 votes
3 answers
234 views

Consistency of diamonds in a concurrent system

I am struggling to find solutions to a problem that I encounter in a concurrent system (I am currently using the Actor Model). I basically have a diamond structure of actors: So D sends messages to C ...
user avatar
  • 149
4 votes
2 answers
2k views

Alternatives to distributed transactions in .NET?

Lately, I've been working on a project which basically is a huge rewrite in .NET Core F# + Event Sourcing + PostgreSQL of an old sub-ledger legacy app written in .NET 4.6 C# + SQL Server. Since the ...
user avatar
  • 309
4 votes
2 answers
3k views

Architecture for message processing with scheduling, at scale

I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with ...
user avatar
  • 143
4 votes
2 answers
690 views

How to implement to flood fill algorithm on multiple machines?

I have a large image distributed over multiple machines for which I need to implement the flood fill algorithm used in MS Paint. I am able to do it with a single machine but what approach must be ...
user avatar
  • 139
4 votes
1 answer
149 views

How are application in separate data centers routed

A bit of down time at work let my mind wonder a bit. I was designing a single location application on Digital Ocean (load balancer to app servers to DB with DNS). All fine. Then I thought what if the ...
user avatar
  • 945
4 votes
2 answers
99 views

Designing an ETL with where there are a few points of entry

I'm trying to think of a scalable solution for my current system. The current system is 3 microscopes 1 processing machine 1. 60-100GB Files come from 2-3 microscopes every 30 minutes 2. That data ...
user avatar
4 votes
1 answer
834 views

distributed application design - using message broker

Background I'm trying to design a layer/component in my distributed application that will communicate between component A and component B. Right now, this "communication" is accomplished by ...
user avatar
  • 479
4 votes
1 answer
2k views

How does Redis (or any typical distributed cache) handle replication conflicts?

Suppose you set up a Redis cluster with one master and two slaves. Two clients are connected to each of the slaves. Both clients make conflicting changes at the same time: What happens if these ...
user avatar
  • 5,375
4 votes
2 answers
112 views

Do coherence and transactional isolation mean the same?

About coherence vs consistency, https://en.wikipedia.org/wiki/Consistency_model says Coherence deals with maintaining a global order in which writes to a single location or single variable are seen ...
user avatar
  • 5,301
4 votes
1 answer
185 views

Is the actor model suited for an application dealing with elapsed time?

I need to create a distributed application with timers / periodically checking two timestamps for elapsed time. Is it possible to achieve following behaviour with the actor model: Spawn Child actor ...
user avatar
  • 49
4 votes
3 answers
348 views

Distributed database algorithm

I have a need for a distributed data store, where existing solutions may not work as the computers these will be running on will be extremely resource limited, for instance 64-128MB RAM. Plus, as a ...
user avatar
  • 149
4 votes
2 answers
89 views

Performance considerations with access control in a distributed system architecture

We're currently working on upgrading a small information system for an enterprise. Currently, the system has a corporate network zone (CORP) where services for internal users are placed, and a data ...
user avatar
  • 181
4 votes
1 answer
250 views

Can an api gateway handle websockets requests besides regular http requests at the same time?

I am working on social platform with a distributed architecture. The platform should offer a chat module, but since the most efficient way to implement a chat is using websockets. I am not sure what ...
user avatar
  • 149
3 votes
3 answers
785 views

Is "Transactional outbox" an anti-pattern?

"Transactional outbox" is a pattern for distributed systems which makes use of a single db transaction to do both, persist changes to the domain model and send a domain event message to the ...
user avatar
  • 523
3 votes
5 answers
1k views

How to resolve cyclic dependencies in Event-driven systems?

Consider two services (bounded contexts by DDD): Sales Billing Sales is responsible for creating orders and Billing for handling payments. Sales tracks orders and Billing holds payments: Sales ...
user avatar
  • 211
3 votes
1 answer
175 views

Should user and service-to-service authentications be separate?

Say I have a system with 5 microservices behind a gateway, and a user signs in through an IDP (OAuth) A user U passes the access token in a request, and the call first reaches the gateway before it ...
user avatar
3 votes
2 answers
963 views

Thinking pragmatically about scalability

I was recently asked as an exercise to design a scalable graph. My first intuition was how to seperate the graph and distribute it (sharding,consistent hashing..etc) Turns out my thinking was on ...
user avatar
  • 133
3 votes
1 answer
151 views

The publish-subscribe pattern: garbage-collecting old subscriptions

I've been studying distributed systems design, in particular the Udi Dahan's class. He talks about the publish-subscribe pattern as a common pattern in messaging-oriented designs. There's obviously a ...
user avatar
  • 715
3 votes
2 answers
160 views

load balancing in shared nothing cluster

I have a system where the load is distributed across the nodes in the cluster and once an entity is assigned to particular node, it is taken to completion through that node only(distribution is saved ...
user avatar
3 votes
1 answer
560 views

Best strategy to flush Redis cache used in a distributed environment

Current setup: We follow the cache-first strategy right now. Always fetch from the cache first. If absent in cache, fetch from DB and update the cache. Cache TTL is 30 min. After this, the cache-key ...
user avatar
3 votes
1 answer
65 views

How to handle changes to local and global representation of an Entity in a distributed environment?

I'm a developer for a retail enterprise IT department, and am currently stuck on the following scenario: Department A is responsible for a CRUD service that's used for handling stores - so when our ...
user avatar
  • 99
3 votes
1 answer
167 views

How to protect an open-source distributed application consisting of clients and servers from forks made by hackers?

TL;DR: How distributed open-source apps like Scuttlebutt are secured from DoS and hackers who can make custom version of application? I'm struggle with designing an open-source distributed application ...
user avatar
3 votes
2 answers
524 views

Architecting a distributed file processing system with leadership election

I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice: Problem : There are a large number of files continuously posted on ...
user avatar
3 votes
1 answer
284 views

Is TCP still considered siginificant overhead compared to UDP? [closed]

I'm working on a distributed application for a graduate class I'm taking. There has been much discussion around implementing a way to maintain peer connection health statuses. Think Gossip. One of the ...
user avatar
  • 173
3 votes
3 answers
1k views

Multiple Databases per Microservice

We have a scenario in which all the important and transactional fields of our business entities are highly structured and relational. The data size of these important fields is also very small. ...
user avatar
  • 139
3 votes
0 answers
83 views

Difficulty understanding how federation increases cache performance for databases?

I am studying system design for distributed systems and in this page (https://github.com/donnemartin/system-design-primer), one of the following advantages was mentioned for federation for databases ...
user avatar
  • 153
3 votes
0 answers
53 views

Release Phase in Lamport's Mutual Exclusion Algorithm

I was studying Lamport's mutual exclusion algorithm from the original paper, and noticed a difference in the release phase to the one given in Wikipedia. In the original paper, author states that : ...
user avatar
  • 181
3 votes
0 answers
66 views

Sharing tasks among parallel independent jobs

I am getting files on HDFS and I have a legacy java app which checks for new files every 15 mins, does some transformations and write them to some remote server. Now, I am facing scaling issue with ...
user avatar
  • 209
3 votes
0 answers
84 views

Architecture and implementation of a de-centralised web service

I'm a web programmer with several years of experience in RoR and node.js. I have a project idea that is quite big and I need some input in how it can be achieved. I heard of the Diaspora project, and ...
user avatar
  • 291
3 votes
0 answers
714 views

Vector clock conflict resolution in event sourced system via local timestamp - is this safe?

We have an occasionally connected, event sourced system: a budgeting application where each client can pull other client's event store from a separate storage (network share, dropbox,...). It is our ...
user avatar
3 votes
1 answer
151 views

Database consistency for a distributed system

I'm working on a phone call monitoring project. The aim is to have one row in the calls table for each ongoing call. Each call instance may be updated by many different servers/threads as more ...
user avatar
  • 141
2 votes
2 answers
368 views

How to discover discovery service

We are designing an architecture of a high availability distibuted system and we are considering server side vs client side service discovery. There is one problem, I need to solve - how is it ...
user avatar
2 votes
1 answer
415 views

Managing consistency in distributed database

I have question regarding system design. Lets say I want to design a Flight booking system ( such as American Airlines). So, consistency when booking a ticket is very important. This booking system is ...
user avatar
2 votes
2 answers
964 views

Decrement counter with high concurrency in distributed system

I am working on problem where a call to reduce a counter will come to a service and if counter is greater zero then call should be able to reduce it else fail. Pretty straightforward? huh! For a ...
user avatar
2 votes
1 answer
91 views

Distributed cart state in microservices

We have legacy app which keeps state of a cart in local storage on a Frontend (this decision was made long time ago by other devs). The cart is cleared from local storage after 2 hours if it is not ...
user avatar
2 votes
2 answers
118 views

Is sequential consistency equivalent to performing memory accesses by a processes in program order and performing each memory access atomically?

In Fundamentals of Parallel Multicore Architecture, by Yan Solihin, p304 defines sequential consistency memory model: Overall, we can express programmers’ implicit expectation of memory access ...
user avatar
  • 5,301