Questions tagged [distributed-system]

This tag can be used by questions on distributed system concept, design, and implementations.

Filter by
Sorted by
Tagged with
1
vote
1answer
31 views

Best strategy to flush Redis cache used in a distributed environment

Current setup: We follow the cache-first strategy right now. Always fetch from the cache first. If absent in cache, fetch from DB and update the cache. Cache TTL is 30 min. After this, the cache-key ...
2
votes
1answer
73 views

Should user and service-to-service authentications be separate?

Say I have a system with 5 microservices behind a gateway, and a user signs in through an IDP (OAuth) A user U passes the access token in a request, and the call first reaches the gateway before it ...
2
votes
0answers
40 views

Automatically extended database-based mutex lock - is it a good idea?

Since I am working in a multi-instance microservice environment, I came accross a problem with making some operations being performed at most by one of the running instances at once. The solution ...
0
votes
0answers
46 views

Realtime report generation and downloads

We host a dashboard where users can view different reports, each powered by a templated SQL query. The users have an option to "download" those reports as CSV. Currently the above use case ...
0
votes
0answers
48 views

Fetching data in separate servers

I have a monolithic REST API and a separate WebSocket server that’s used for real-time updates. They're both separate so that I can scale them independently. The API fetches data from the database and ...
0
votes
0answers
31 views

How to share dynamic configuration data in a clustered environment?

I'm working on an side-project which will perform some API requests to a third party service. I'm limited how many requests I can send, so each minute the application is scheduled to get a certain ...
0
votes
0answers
30 views

How to decide whether to spawn more worker instances in a node according to bandwidth utilization

We're working on a distributed software which transfers data around. The software is composed by a central server application (called "the server") and several workers which synchronize with ...
0
votes
1answer
74 views

How can message queues improve scalability?

I have read in a DDD book that using message queues between communicating services can make the whole architecture more scalable, amazon's documentation mentions that queues provide granular ...
1
vote
1answer
127 views

What performance benefits does a Redis cluster have over having all machines connect to a single Redis node? [closed]

Suppose I have data that easily fits in memory in a redis database. The redis server runs as a single service on a single process on a single machine. Let's say I have a 100 different machines connect ...
3
votes
3answers
435 views

Is “Transactional outbox” an anti-pattern?

"Transactional outbox" is a pattern for distributed systems which makes use of a single db transaction to do both, persist changes to the domain model and send a domain event message to the ...
0
votes
0answers
31 views

Different Ways to implement validation registry

Task To Accomplish I am creating a validation service which validates a user profile data for correctness. A user profile can have following information: Name Address Social Security Number(SSN) ...
1
vote
3answers
130 views

Which code to copy-paste and which to centralize?

In a fairly complex system with a dozen or more git repositories, most of them are done in a cookie-cutter fashion. What is or would be your best approach to deal with duplication? In my opinion the ...
2
votes
2answers
83 views

Is sequential consistency equivalent to performing memory accesses by a processes in program order and performing each memory access atomically?

In Fundamentals of Parallel Multicore Architecture, by Yan Solihin, p304 defines sequential consistency memory model: Overall, we can express programmers’ implicit expectation of memory access ...
0
votes
0answers
34 views

Pubsub message design in microservices

Example We're designing a social network, and when the user logs in, we need to authenticate them and then load their feed. In order to do this, the client first makes a HTTPS request to the server, ...
4
votes
2answers
103 views

Do coherence and transactional isolation mean the same?

About coherence vs consistency, https://en.wikipedia.org/wiki/Consistency_model says Coherence deals with maintaining a global order in which writes to a single location or single variable are seen ...
2
votes
1answer
62 views

How to handle changes to local and global representation of an Entity in a distributed environment?

I'm a developer for a retail enterprise IT department, and am currently stuck on the following scenario: Department A is responsible for a CRUD service that's used for handling stores - so when our ...
3
votes
1answer
141 views

Database consistency for a distributed system

I'm working on a phone call monitoring project. The aim is to have one row in the calls table for each ongoing call. Each call instance may be updated by many different servers/threads as more ...
28
votes
5answers
6k views

I'm not seeing 'tightly coupled code' as one of the drawbacks of a monolithic application architecture

One of the most common things I see when discussing pros/cons of microservice vs monolithic architecture is that monolithic applications have, or always trend toward, 'tight coupling.' To be honest, I'...
0
votes
2answers
253 views

Why is it considered hard to maintain strong consistency in a distributed system?

Why can't you just use strongly consistent reads for all your DB reads, with retries on 500 responses? According to CAP theorem increasing consistency should probably lower availability, but can't the ...
5
votes
1answer
101 views

Architecting basket in high scale distributed application

I'm designing an e-commerce application. Main flow is pretty straightforward: customer add items to basket, checkouts the basket (place an order) and waits for delivery. There are following ...
1
vote
1answer
70 views

Can passive and active replication handle byzantine failures?

Currently studying about replication and I was wondering: In Passive replication, we have an agreement stage where the Primary replica manager (RM) waits for ack from the backup RM, wouldn't it be ...
1
vote
0answers
112 views

How does Stackoverflow comment system work

When someone comments on my question/answer/comment I will see the notification from the inbox, and it also shows under my response tab. Later when the comment is edited, the comment shown under the ...
6
votes
2answers
2k views

What is the meaning of fan-out

I see this word in many places but don't get it. From the Wikipedia, In message-oriented middleware solutions, fan-out is a messaging pattern used to model an information exchange that implies the ...
3
votes
1answer
158 views

How to protect an open-source distributed application consisting of clients and servers from forks made by hackers?

TL;DR: How distributed open-source apps like Scuttlebutt are secured from DoS and hackers who can make custom version of application? I'm struggle with designing an open-source distributed application ...
1
vote
1answer
73 views

Should I use ROA or SOA for a distributed application and how could I implement REST If using ROA

For my final year project I'm looking to build a distributed version of a popular benchmarking client (this has already been done using various methods involving some form of existing frameworks), I ...
4
votes
2answers
97 views

Designing an ETL with where there are a few points of entry

I'm trying to think of a scalable solution for my current system. The current system is 3 microscopes 1 processing machine 1. 60-100GB Files come from 2-3 microscopes every 30 minutes 2. That data ...
4
votes
3answers
278 views

How can adding redunancy adversely affect performance

I have been reading about system design and was going through this popular article - A Word on Scalability The author here says: An always-on service is said to be scalable if adding resources to ...
2
votes
0answers
55 views

SAGA pattern - how to handle situation when some message is not received by orchestrator

Recently I'm studying Saga pattern, but I'm struggling to imagine how it will work in real system. Standard example is making an order, which consist of reserving flight, hotel and a car. We can ...
1
vote
1answer
151 views

Distributed message queue, propagating queue leader/follower information

I'm designing a distributed queue message queue and I'm not sure how to get the leader information to another component. This is my current design: I have the following components: FrontEnd: request ...
0
votes
1answer
160 views

What should we consider when designing a load balancer with minimum cost

Given a set of processes running on a cluster of hosts, I'm designing a system that load balances the hosts through live migrations (carried out by an external service) of the processes. The goal of ...
0
votes
1answer
39 views

Distributed processing of a large number of queues

I have a rather basic application hosted on Kubernetes, which connects to a Mongo database. The app has a wallet feature. A user can put money in their wallet using real-world payments (e.g. via ...
2
votes
1answer
65 views

Distributed cart state in microservices

We have legacy app which keeps state of a cart in local storage on a Frontend (this decision was made long time ago by other devs). The cart is cleared from local storage after 2 hours if it is not ...
0
votes
1answer
181 views

Managing consistency in distributed database

I have question regarding system design. Lets say I want to design a Flight booking system ( such as American Airlines). So, consistency when booking a ticket is very important. This booking system is ...
0
votes
1answer
46 views

System design to store pieces of data with specific ttyl to then refetch

I have a system which generates pieces of data that needs to be tracked and assigned a specific TTYL value (a value in seconds) for how long they are supposed to live before being re-processed. I ...
-2
votes
2answers
109 views

How can one choose between consistency and availability if the system you are designing is Tiny URL

I am putting together a system design for something similar to Tiny URL. It provides an API for mapping a URL to a short URL. Once created the short URL can be used to access the original URL. This ...
1
vote
1answer
96 views

Implemeting a distributed n-Producers/1-Consumer service for critical-mission system

I am trying to implement a distributed version of multi-producer/1-consumer for a critical-mission system. I'm looking for good alternatives to the current approach based on RDBMS. The problem The ...
3
votes
4answers
501 views

If a system talks to a database to get some previous information to serve a request, does that make the system **stateful** or **stateless**?

I've read this example which basically states that, since a method has to hold a single private field.. It is considered a stateful system. So basically ANYTHING system that calls upon some sort of ...
1
vote
2answers
147 views

URL shortening : data partitioning

I've been looking in many places about this question - most top Google search results are copy-pasted from a single source, and others are not particularly helpful. Not sure if I am allowed to include ...
2
votes
1answer
122 views

What architecture/design pattern for sharing data between several people, without centralization?

Let's say my software lets you place a square on the map and say "This is a good place to find Pokemon". I need this data to go to my gym leader's map, and to my friend's copy of Pokemon Go (...
2
votes
1answer
227 views

What are the different messaging patterns for multiple producers to single consumer through a distributed queue?

We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
0
votes
0answers
29 views

Experiences with coarse grained and fine grained message designs in an ECST pattern

I am currently facing a dilemma in the design of a message oriented distributed system following an Event Carried State Transfer pattern. This is essentially a "near real time ETL" used to ...
9
votes
2answers
314 views

When bounded contexts and “microservices” collide. A distributed systems dilemma in diagram form

You can't please everyone. Some people want a lot of context and background on sites like this. Others do not. If you don't want the background, skip the first three paragraphs. I am a software ...
2
votes
1answer
110 views

The publish-subscribe pattern: garbage-collecting old subscriptions

I've been studying distributed systems design, in particular the Udi Dahan's class. He talks about the publish-subscribe pattern as a common pattern in messaging-oriented designs. There's obviously a ...
1
vote
1answer
85 views

When keeping all state in an SQL DB, how to handle one-off variables?

I noticed this pattern in my programs: I'm building a system, like the one I'm building for my local hackerspace right now. It's using database as its "brain" - a scraper as a separate ...
3
votes
3answers
321 views

How does a CDN get “data” when there's hundreds of terabytes stored in data centers?

I am still trying to wrap my head around CDN's in a real world scenario. Suppose I am building a Netflix clone. I have about 1,000 terabytes of video content stored in an S3 bucket. Since Netflix ...
3
votes
2answers
857 views

Alternatives to distributed transactions in .NET?

Lately, I've been working on a project which basically is a huge rewrite in .NET Core F# + Event Sourcing + PostgreSQL of an old sub-ledger legacy app written in .NET 4.6 C# + SQL Server. Since the ...
0
votes
1answer
92 views

Maintaining Objects Across API Deployment Instances

I am working on a web application as a hobby and trying to learn some concepts related to cloud development and distributed applications. I am currently targeting an AWS EC2 instance as a deployment ...
-1
votes
1answer
103 views

Facebook sharding by user_id or by photo_id?

I'm reading a design suggestion for facebook/instagram social network in grokking the system interview (closed content :( so I'll describe the relevant part in my question). They are speculating over ...
1
vote
1answer
82 views

Is weak consistency still relevant in systems that record timestamps?

There are multiple aspects and quality that are desirable for a distributed system. Among these are consistency and availability. According to the CAP theorem, if partition tolerance is a must, the ...
-1
votes
2answers
60 views

Concurrency issue in job-based system

Setup Say I have a database table called Events which contains events for when a user becomes active or inactive EventId|Timestamp |User|Status | -------|--------------|----|--------| 1 |...