Questions tagged [distributed-system]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
7
votes
1answer
188 views

When bounded contexts and “microservices” collide. A distributed systems dilemma in diagram form

You can't please everyone. Some people want a lot of context and background on sites like this. Others do not. If you don't want the background, skip the first three paragraphs. I am a software ...
2
votes
1answer
63 views

The publish-subscribe pattern: garbage-collecting old subscriptions

I've been studying distributed systems design, in particular the Udi Dahan's class. He talks about the publish-subscribe pattern as a common pattern in messaging-oriented designs. There's obviously a ...
1
vote
1answer
79 views

When keeping all state in an SQL DB, how to handle one-off variables?

I noticed this pattern in my programs: I'm building a system, like the one I'm building for my local hackerspace right now. It's using database as its "brain" - a scraper as a separate ...
0
votes
0answers
16 views

Messaging - Responsibility of Consumer vs Publisher for Active-Active failover

Say there is an external client-DC1/DC2, posting requests to a service in DC1 and DC2 respectively. Now, if DC1 is down, should the external client-DC1 responsible for routing the request message to ...
3
votes
3answers
269 views

How does a CDN get “data” when there's hundreds of terabytes stored in data centers?

I am still trying to wrap my head around CDN's in a real world scenario. Suppose I am building a Netflix clone. I have about 1,000 terabytes of video content stored in an S3 bucket. Since Netflix ...
2
votes
2answers
109 views

Alternatives to distributed transactions in .NET?

Lately, I've been working on a project which basically is a huge rewrite in .NET Core F# + Event Sourcing + PostgreSQL of an old sub-ledger legacy app written in .NET 4.6 C# + SQL Server. Since the ...
0
votes
0answers
17 views

Maintaining Objects Across API Deployment Instances

I am working on a web application as a hobby and trying to learn some concepts related to cloud development and distributed applications. I am currently targeting an AWS EC2 instance as a deployment ...
-2
votes
1answer
39 views

facebook sharding by user_id or by photo_id?

I'm reading a design suggestion for facebook/instagram social network in grokking the system interview (closed content :( so I'll describe the relevant part in my question). They are speculating over ...
1
vote
1answer
56 views

Is weak consistency still relevant in systems that record timestamps?

There are multiple aspects and quality that are desirable for a distributed system. Among these are consistency and availability. According to the CAP theorem, if partition tolerance is a must, the ...
-1
votes
2answers
56 views

Concurrency issue in job-based system

Setup Say I have a database table called Events which contains events for when a user becomes active or inactive EventId|Timestamp |User|Status | -------|--------------|----|--------| 1 |...
1
vote
2answers
71 views

Synchronous communcation vs caching in microservices

I'm trying to build a distributed system and have been researching how best to manage dependencies between services. The common example in tutorials is an ordering system - let's say I have a Catalog, ...
2
votes
4answers
195 views

How to resolve cyclic dependencies in Event-driven systems?

Consider two services (bounded contexts by DDD): Sales Billing Sales is responsible for creating orders and Billing for handling payments. Sales tracks orders and Billing holds payments: Sales ...
-3
votes
1answer
254 views

Why can't we vertically scale a system infinitely?

When talking about scaling a system, it's often said that vertical scaling has limitations. So after a point, we need to scale the system horizontally. What are the limitations of: Main memory - ...
1
vote
0answers
23 views

Do Multi-Primary replicated distributed systems guarantee sequential consistency?

I know that Primary-backup replicated distributed systems guarantee sequential consistency. My question is whether multi-primary systems also achieve that. I mean if they use consensus (i.e: Paxos ...
0
votes
1answer
57 views

Need to test for equality on vector clocks

I've heard about vector clocks and how to test if a message was sent before another message. E.g. Message A was sent before message B if every element of the vector of message A is smaller or equal ...
-1
votes
2answers
390 views

Race condition in Distributed Systems

We have a set of micro-services that are functionally dependent and have intertwined validation logic. e.g. consider 'Credit-Card' service and 'Loan' service. When user has a credit card then loan ...
-4
votes
1answer
145 views

What difference and relation are between fault tolerance and (high) availability?

From Coulouris' Distributed Systems 5ed Chapter 18 Replication 18.1 Introduction Increased availability: Users require services to be highly available. That is, the proportion ...
-3
votes
2answers
110 views

How does a distributed system both tolerate network partition and achieve consistency?

In the CAP theorem, there are only three possible cases: C and A without P C and P without A A and P without C How can we have both P and C in the second case? Doesn't propagation of update from ...
-5
votes
1answer
59 views

Does stale data due to weak level of consistency count as Byzantine failure?

I have difficulty understand Section 18.3 Fault Tolerance Services under Ch18 Replication in Coulouris' Distributed Systems. If my reading and understanding is correct (which might not), Section 18....
-4
votes
2answers
77 views

What are distributed transactions supposed to satisfy?

I am lost when reading Design Data Intensive Applications. In a nondistributed database system, nondistributed transactions are to satisfy ACID. In a distributed data store, what are distributed ...
-3
votes
2answers
147 views

What is the difference between masking and tolerating failures?

Distributed Systems 5ed by Coulouris says on p21-22 1.5.5 Failure handling Detecting failures: Some failures can be detected. For example, checksums can be used to detect corrupted data in a ...
-2
votes
1answer
79 views

In the distributed computing problem known as the “Byzantine fault”, what differences and relations are between commander, lieutenant, and general? [closed]

Distributed systems by Coulouris says The Byzantine generals problem • In the informal statement of the Byzantine generals problem [Lamport et al. 1982], three or more generals are to agree to ...
0
votes
0answers
11 views

Distribute processing load across multiple nodes of same Web Application

We have an application deployed across multiple nodes. There is a scheduled job inside this application that would be run on any one of the nodes. When the job is triggered it is supposed to read ...
0
votes
0answers
392 views

log centralization and central aggregation system design

I'm thinking of how would I go about creating my own logging centralization and centralized aggregation system. Imagined system has 10 - 100 nodes. Each node runs a LAMP(Linux, Apache, MySql, PHP) ...
0
votes
3answers
42 views

Offline Synchronisation with Changing Access Rights

We got a distributed application where a client syncs certain data portions to its offline storage and performs modifications on it. The data is structured in container entities "Container" and ...
0
votes
1answer
74 views

Are distributed file systems and naming services middleware?

From Distributed Systems by Coulours, a distributed system is partitioned into layers Middleware is defined as Middleware was defined in Section 1.5.1 as a layer of software whose purpose is to ...
0
votes
3answers
72 views

Is shared disk architecture scaling up or scaling out?

In Design Data Intensive Applications, If all you need is to scale to higher load, the simplest approach is to buy a more powerful machine (sometimes called vertical scaling or scaling up). ...
0
votes
1answer
179 views

Should microservices manage their own media file logics?

I've heard that when you designing a microservice, the service should includes everything it needs to make them loose coupling instead of calling each other (services). Based on this idea, should ...
1
vote
0answers
25 views

Consistency and Availability in distributed hashing Key value store

During the last semester of my college, we were asked to read the (chord paper) and implement it without fault tolerance. We built it with the following feature: A new node can join. A client can ...
3
votes
3answers
217 views

Multiple Databases per Microservice

We have a scenario in which all the important and transactional fields of our business entities are highly structured and relational. The data size of these important fields is also very small. ...
2
votes
0answers
48 views

Distributing tasks to several clients

I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements. I have 1-...
0
votes
0answers
17 views

Turn based access to update shared resource such as inventory

I am trying to design a system where a Product can only be sold while inventory is available. This is an e-commerce platform where users can buy specific products with designated inventory. A similar ...
3
votes
1answer
145 views

Design question on synchronization of two asynchronous data streams

I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
0
votes
1answer
40 views

How to design a topped-up internal balance for a replicated API?

My product is an API which charges an amount of USD every N requests. I would like to allow my users to top-up their internal balances through PayPal, then decrease their balances when they use the ...
0
votes
0answers
88 views

What is a good design for maintaining consistency in a last write wins database without event sourcing

We have an application which on every resource update(resource update, create, delete)creates a message and puts it on a queue. This message is picked up by a worker process, converted to a format ...
0
votes
0answers
42 views

How should the interaction between the MIS instances of a large system happen?

Overall situations: The plan was to build a large system (Management Information System or MIS type) to be used throughout the country, by say 100,000 users daily. The team decided to split that into ...
0
votes
0answers
27 views

How to arrange synchronicity reporting in an eventually consistent, distributed system?

Context: A typical corporate landscape with many systems located in different data centers, with firewalls, without uniformised interfacing. Some data X is being replicated for availability and ...
1
vote
2answers
254 views

How to discover discovery service

We are designing an architecture of a high availability distibuted system and we are considering server side vs client side service discovery. There is one problem, I need to solve - how is it ...
1
vote
1answer
628 views

Architecture for message processing with scheduling, at scale

I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with ...
1
vote
1answer
96 views

Would this system be considered decentralized or distributed?

I am currently working on a report regarding a matrix multiplication distribution system I have been working on. But after reading a lot of somewhat contradictory sources online I have become in doubt ...
-1
votes
1answer
118 views

Documenting a distributed application [closed]

Recently, I've became the integration lead for a large distributed system. I have two main challenges before I can take on anything else: first document all system interfaces and it's relationships ...
0
votes
0answers
42 views

Do web companies use the master replica in data center nearest to the user to do writes?

I am interested how multi datacenter (multi homing in Google terms) systems handle writes to strongly consistent user data? Is the master replica for each user placed in the same datacenter and writes ...
1
vote
1answer
33 views

How does Operational Transformations work when offering offline editing capabilities?

As far as I understand, OT works really well with concurrent editing. There is only one remote copy of the document, and all editors update that remote copy simultaneously. There is a concept of lag ...
0
votes
1answer
146 views

Distribute jobs evenly among a set of servers

I have the need to distribute a set of long running jobs across a cluster of containers in ECS. These jobs essentially would need to open a socket connection with a remote server and begin streaming ...
0
votes
1answer
268 views

Robust locking mechanism between n service instances

I have a job that runs every hour. There can be n identical Node.js microservices running in a stack. The job must be executed only once per hour, though, so I've implemented the following locking ...
1
vote
0answers
50 views

Develop distributed system contain application with poor architecture [duplicate]

In last months the business continues dynamic grow so we must develop new features along distributed system. Unfortunately we do not have much time to deliver new solutions. The application is ...
1
vote
2answers
252 views

Designing a big data web app

How do you design a website that allows users to query a large amount of user data, more specifically: there are ~100 million users with ~100TB of data, data is stored in HDFS (not a database) number ...
5
votes
2answers
1k views

How to prevent overlap booking on a calendar booking system

As an exercise, I am trying to design a simple calendar booking system for multiple meeting rooms. I kind of got my head around some requirements such as find available rooms for a given time range, ...
2
votes
1answer
369 views

Configuration in a Distributed Application

I am developing a distributed application built as a collection of separate services which are served from multiple load balanced instances and trying to wrap my head around the distributed ...
2
votes
2answers
664 views

Message Broker in Client-Server-Applications

I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ). A simplified diagram of my ...