Questions tagged [distributed-system]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0
votes
0answers
16 views

Idempotent Kafka consumer with an “eventually consistent” database

What are my options if I want to make my Kafka consumer idempotent (due to Kafka's at-least-once semantics) if my project's only database is DynamoDB, which is eventually consistent? I'm afraid ...
0
votes
1answer
22 views

Distributed processing of a large number of queues

I have a rather basic application hosted on Kubernetes, which connects to a Mongo database. The app has a wallet feature. A user can put money in their wallet using real-world payments (e.g. via ...
2
votes
1answer
61 views

Distributed cart state in microservices

We have legacy app which keeps state of a cart in local storage on a Frontend (this decision was made long time ago by other devs). The cart is cleared from local storage after 2 hours if it is not ...
0
votes
1answer
88 views

Managing consistency in distributed database

I have question regarding system design. Lets say I want to design a Flight booking system ( such as American Airlines). So, consistency when booking a ticket is very important. This booking system is ...
0
votes
1answer
45 views

System design to store pieces of data with specific ttyl to then refetch

I have a system which generates pieces of data that needs to be tracked and assigned a specific TTYL value (a value in seconds) for how long they are supposed to live before being re-processed. I ...
-3
votes
2answers
92 views

How can one choose between consistency and availability if the system you are designing is Tiny URL

I am putting together a system design for something similar to Tiny URL. It provides an API for mapping a URL to a short URL. Once created the short URL can be used to access the original URL. This ...
1
vote
1answer
91 views

Implemeting a distributed n-Producers/1-Consumer service for critical-mission system

I am trying to implement a distributed version of multi-producer/1-consumer for a critical-mission system. I'm looking for good alternatives to the current approach based on RDBMS. The problem The ...
3
votes
4answers
479 views

If a system talks to a database to get some previous information to serve a request, does that make the system **stateful** or **stateless**?

I've read this example which basically states that, since a method has to hold a single private field.. It is considered a stateful system. So basically ANYTHING system that calls upon some sort of ...
1
vote
2answers
120 views

URL shortening : data partitioning

I've been looking in many places about this question - most top Google search results are copy-pasted from a single source, and others are not particularly helpful. Not sure if I am allowed to include ...
2
votes
1answer
117 views

What architecture/design pattern for sharing data between several people, without centralization?

Let's say my software lets you place a square on the map and say "This is a good place to find Pokemon". I need this data to go to my gym leader's map, and to my friend's copy of Pokemon Go (...
2
votes
1answer
98 views

What are the different messaging patterns for multiple producers to single consumer through a distributed queue?

We have multiple producers that publish messages to the same SQS queue. We have a single consumer that processes the messages. The producers do not care about the response. It is more like a broadcast ...
0
votes
0answers
21 views

Experiences with coarse grained and fine grained message designs in an ECST pattern

I am currently facing a dilemma in the design of a message oriented distributed system following an Event Carried State Transfer pattern. This is essentially a "near real time ETL" used to ...
7
votes
1answer
225 views

When bounded contexts and “microservices” collide. A distributed systems dilemma in diagram form

You can't please everyone. Some people want a lot of context and background on sites like this. Others do not. If you don't want the background, skip the first three paragraphs. I am a software ...
2
votes
1answer
74 views

The publish-subscribe pattern: garbage-collecting old subscriptions

I've been studying distributed systems design, in particular the Udi Dahan's class. He talks about the publish-subscribe pattern as a common pattern in messaging-oriented designs. There's obviously a ...
1
vote
1answer
83 views

When keeping all state in an SQL DB, how to handle one-off variables?

I noticed this pattern in my programs: I'm building a system, like the one I'm building for my local hackerspace right now. It's using database as its "brain" - a scraper as a separate ...
0
votes
0answers
16 views

Messaging - Responsibility of Consumer vs Publisher for Active-Active failover

Say there is an external client-DC1/DC2, posting requests to a service in DC1 and DC2 respectively. Now, if DC1 is down, should the external client-DC1 responsible for routing the request message to ...
3
votes
3answers
285 views

How does a CDN get “data” when there's hundreds of terabytes stored in data centers?

I am still trying to wrap my head around CDN's in a real world scenario. Suppose I am building a Netflix clone. I have about 1,000 terabytes of video content stored in an S3 bucket. Since Netflix ...
3
votes
2answers
162 views

Alternatives to distributed transactions in .NET?

Lately, I've been working on a project which basically is a huge rewrite in .NET Core F# + Event Sourcing + PostgreSQL of an old sub-ledger legacy app written in .NET 4.6 C# + SQL Server. Since the ...
0
votes
1answer
65 views

Maintaining Objects Across API Deployment Instances

I am working on a web application as a hobby and trying to learn some concepts related to cloud development and distributed applications. I am currently targeting an AWS EC2 instance as a deployment ...
-2
votes
1answer
40 views

facebook sharding by user_id or by photo_id?

I'm reading a design suggestion for facebook/instagram social network in grokking the system interview (closed content :( so I'll describe the relevant part in my question). They are speculating over ...
1
vote
1answer
59 views

Is weak consistency still relevant in systems that record timestamps?

There are multiple aspects and quality that are desirable for a distributed system. Among these are consistency and availability. According to the CAP theorem, if partition tolerance is a must, the ...
-1
votes
2answers
56 views

Concurrency issue in job-based system

Setup Say I have a database table called Events which contains events for when a user becomes active or inactive EventId|Timestamp |User|Status | -------|--------------|----|--------| 1 |...
2
votes
2answers
102 views

Synchronous communcation vs caching in microservices

I'm trying to build a distributed system and have been researching how best to manage dependencies between services. The common example in tutorials is an ordering system - let's say I have a Catalog, ...
2
votes
4answers
273 views

How to resolve cyclic dependencies in Event-driven systems?

Consider two services (bounded contexts by DDD): Sales Billing Sales is responsible for creating orders and Billing for handling payments. Sales tracks orders and Billing holds payments: Sales ...
-3
votes
1answer
269 views

Why can't we vertically scale a system infinitely?

When talking about scaling a system, it's often said that vertical scaling has limitations. So after a point, we need to scale the system horizontally. What are the limitations of: Main memory - ...
1
vote
0answers
23 views

Do Multi-Primary replicated distributed systems guarantee sequential consistency?

I know that Primary-backup replicated distributed systems guarantee sequential consistency. My question is whether multi-primary systems also achieve that. I mean if they use consensus (i.e: Paxos ...
0
votes
1answer
62 views

Need to test for equality on vector clocks

I've heard about vector clocks and how to test if a message was sent before another message. E.g. Message A was sent before message B if every element of the vector of message A is smaller or equal ...
-1
votes
2answers
583 views

Race condition in Distributed Systems

We have a set of micro-services that are functionally dependent and have intertwined validation logic. e.g. consider 'Credit-Card' service and 'Loan' service. When user has a credit card then loan ...
-4
votes
1answer
180 views

What difference and relation are between fault tolerance and (high) availability?

From Coulouris' Distributed Systems 5ed Chapter 18 Replication 18.1 Introduction Increased availability: Users require services to be highly available. That is, the proportion ...
-3
votes
2answers
146 views

How does a distributed system both tolerate network partition and achieve consistency?

In the CAP theorem, there are only three possible cases: C and A without P C and P without A A and P without C How can we have both P and C in the second case? Doesn't propagation of update from ...
-5
votes
1answer
59 views

Does stale data due to weak level of consistency count as Byzantine failure?

I have difficulty understand Section 18.3 Fault Tolerance Services under Ch18 Replication in Coulouris' Distributed Systems. If my reading and understanding is correct (which might not), Section 18....
-4
votes
2answers
86 views

What are distributed transactions supposed to satisfy?

I am lost when reading Design Data Intensive Applications. In a nondistributed database system, nondistributed transactions are to satisfy ACID. In a distributed data store, what are distributed ...
-3
votes
2answers
154 views

What is the difference between masking and tolerating failures?

Distributed Systems 5ed by Coulouris says on p21-22 1.5.5 Failure handling Detecting failures: Some failures can be detected. For example, checksums can be used to detect corrupted data in a ...
-2
votes
1answer
81 views

In the distributed computing problem known as the “Byzantine fault”, what differences and relations are between commander, lieutenant, and general? [closed]

Distributed systems by Coulouris says The Byzantine generals problem • In the informal statement of the Byzantine generals problem [Lamport et al. 1982], three or more generals are to agree to ...
0
votes
0answers
13 views

Distribute processing load across multiple nodes of same Web Application

We have an application deployed across multiple nodes. There is a scheduled job inside this application that would be run on any one of the nodes. When the job is triggered it is supposed to read ...
0
votes
0answers
445 views

log centralization and central aggregation system design

I'm thinking of how would I go about creating my own logging centralization and centralized aggregation system. Imagined system has 10 - 100 nodes. Each node runs a LAMP(Linux, Apache, MySql, PHP) ...
0
votes
3answers
46 views

Offline Synchronisation with Changing Access Rights

We got a distributed application where a client syncs certain data portions to its offline storage and performs modifications on it. The data is structured in container entities "Container" and ...
0
votes
1answer
102 views

Are distributed file systems and naming services middleware?

From Distributed Systems by Coulours, a distributed system is partitioned into layers Middleware is defined as Middleware was defined in Section 1.5.1 as a layer of software whose purpose is to ...
0
votes
3answers
87 views

Is shared disk architecture scaling up or scaling out?

In Design Data Intensive Applications, If all you need is to scale to higher load, the simplest approach is to buy a more powerful machine (sometimes called vertical scaling or scaling up). ...
0
votes
1answer
258 views

Should microservices manage their own media file logics?

I've heard that when you designing a microservice, the service should includes everything it needs to make them loose coupling instead of calling each other (services). Based on this idea, should ...
1
vote
0answers
29 views

Consistency and Availability in distributed hashing Key value store

During the last semester of my college, we were asked to read the (chord paper) and implement it without fault tolerance. We built it with the following feature: A new node can join. A client can ...
3
votes
3answers
326 views

Multiple Databases per Microservice

We have a scenario in which all the important and transactional fields of our business entities are highly structured and relational. The data size of these important fields is also very small. ...
2
votes
0answers
49 views

Distributing tasks to several clients

I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements. I have 1-...
0
votes
0answers
18 views

Turn based access to update shared resource such as inventory

I am trying to design a system where a Product can only be sold while inventory is available. This is an e-commerce platform where users can buy specific products with designated inventory. A similar ...
3
votes
1answer
180 views

Design question on synchronization of two asynchronous data streams

I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
0
votes
1answer
46 views

How to design a topped-up internal balance for a replicated API?

My product is an API which charges an amount of USD every N requests. I would like to allow my users to top-up their internal balances through PayPal, then decrease their balances when they use the ...
0
votes
0answers
115 views

What is a good design for maintaining consistency in a last write wins database without event sourcing

We have an application which on every resource update(resource update, create, delete)creates a message and puts it on a queue. This message is picked up by a worker process, converted to a format ...
0
votes
0answers
42 views

How should the interaction between the MIS instances of a large system happen?

Overall situations: The plan was to build a large system (Management Information System or MIS type) to be used throughout the country, by say 100,000 users daily. The team decided to split that into ...
0
votes
0answers
29 views

How to arrange synchronicity reporting in an eventually consistent, distributed system?

Context: A typical corporate landscape with many systems located in different data centers, with firewalls, without uniformised interfacing. Some data X is being replicated for availability and ...
1
vote
2answers
255 views

How to discover discovery service

We are designing an architecture of a high availability distibuted system and we are considering server side vs client side service discovery. There is one problem, I need to solve - how is it ...