Questions tagged [distributed-system]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
20 views

Do Multi-Primary replicated distributed systems guarantee sequential consistency?

I know that Primary-backup replicated distributed systems guarantee sequential consistency. My question is whether multi-primary systems also achieve that. I mean if they use consensus (i.e: Paxos ...
0
votes
1answer
52 views

Need to test for equality on vector clocks

I've heard about vector clocks and how to test if a message was sent before another message. E.g. Message A was sent before message B if every element of the vector of message A is smaller or equal ...
-1
votes
2answers
159 views

Race condition in Distributed Systems

We have a set of micro-services that are functionally dependent and have intertwined validation logic. e.g. consider 'Credit-Card' service and 'Loan' service. When user has a credit card then loan ...
-4
votes
1answer
78 views

What difference and relation are between fault tolerance and (high) availability?

From Coulouris' Distributed Systems 5ed Chapter 18 Replication 18.1 Introduction Increased availability: Users require services to be highly available. That is, the proportion ...
-3
votes
2answers
95 views

How does a distributed system both tolerate network partition and achieve consistency?

In the CAP theorem, there are only three possible cases: C and A without P C and P without A A and P without C How can we have both P and C in the second case? Doesn't propagation of update from ...
-5
votes
1answer
56 views

Does stale data due to weak level of consistency count as Byzantine failure?

I have difficulty understand Section 18.3 Fault Tolerance Services under Ch18 Replication in Coulouris' Distributed Systems. If my reading and understanding is correct (which might not), Section 18....
-4
votes
2answers
73 views

What are distributed transactions supposed to satisfy?

I am lost when reading Design Data Intensive Applications. In a nondistributed database system, nondistributed transactions are to satisfy ACID. In a distributed data store, what are distributed ...
-3
votes
2answers
145 views

What is the difference between masking and tolerating failures?

Distributed Systems 5ed by Coulouris says on p21-22 1.5.5 Failure handling Detecting failures: Some failures can be detected. For example, checksums can be used to detect corrupted data in a ...
-2
votes
1answer
78 views

In the distributed computing problem known as the “Byzantine fault”, what differences and relations are between commander, lieutenant, and general? [closed]

Distributed systems by Coulouris says The Byzantine generals problem • In the informal statement of the Byzantine generals problem [Lamport et al. 1982], three or more generals are to agree to ...
0
votes
0answers
10 views

Distribute processing load across multiple nodes of same Web Application

We have an application deployed across multiple nodes. There is a scheduled job inside this application that would be run on any one of the nodes. When the job is triggered it is supposed to read ...
0
votes
0answers
30 views

log centralization and central aggregation system design

I'm thinking of how would I go about creating my own logging centralization and centralized aggregation system. Imagined system has 10 - 100 nodes. Each node runs a LAMP(Linux, Apache, MySql, PHP) ...
0
votes
3answers
42 views

Offline Synchronisation with Changing Access Rights

We got a distributed application where a client syncs certain data portions to its offline storage and performs modifications on it. The data is structured in container entities "Container" and ...
0
votes
1answer
52 views

Are distributed file systems and naming services middleware?

From Distributed Systems by Coulours, a distributed system is partitioned into layers Middleware is defined as Middleware was defined in Section 1.5.1 as a layer of software whose purpose is to ...
0
votes
3answers
59 views

Is shared disk architecture scaling up or scaling out?

In Design Data Intensive Applications, If all you need is to scale to higher load, the simplest approach is to buy a more powerful machine (sometimes called vertical scaling or scaling up). ...
0
votes
1answer
88 views

Should microservices manage their own media file logics?

I've heard that when you designing a microservice, the service should includes everything it needs to make them loose coupling instead of calling each other (services). Based on this idea, should ...
1
vote
0answers
18 views

Consistency and Availability in distributed hashing Key value store

During the last semester of my college, we were asked to read the (chord paper) and implement it without fault tolerance. We built it with the following feature: A new node can join. A client can ...
3
votes
3answers
149 views

Multiple Databases per Microservice

We have a scenario in which all the important and transactional fields of our business entities are highly structured and relational. The data size of these important fields is also very small. ...
2
votes
0answers
40 views

Distributing tasks to several clients

I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements. I have 1-...
0
votes
0answers
16 views

Turn based access to update shared resource such as inventory

I am trying to design a system where a Product can only be sold while inventory is available. This is an e-commerce platform where users can buy specific products with designated inventory. A similar ...
2
votes
1answer
119 views

Design question on synchronization of two asynchronous data streams

I have two async streams suppose- Trip : {tripId, date, city} Bill : {billId, tripId, date, amount}. I need to design a system to get real time aggregated view of following nature: City, TripCount, ...
0
votes
1answer
37 views

How to design a topped-up internal balance for a replicated API?

My product is an API which charges an amount of USD every N requests. I would like to allow my users to top-up their internal balances through PayPal, then decrease their balances when they use the ...
0
votes
0answers
65 views

What is a good design for maintaining consistency in a last write wins database without event sourcing

We have an application which on every resource update(resource update, create, delete)creates a message and puts it on a queue. This message is picked up by a worker process, converted to a format ...
0
votes
0answers
42 views

How should the interaction between the MIS instances of a large system happen?

Overall situations: The plan was to build a large system (Management Information System or MIS type) to be used throughout the country, by say 100,000 users daily. The team decided to split that into ...
0
votes
0answers
26 views

How to arrange synchronicity reporting in an eventually consistent, distributed system?

Context: A typical corporate landscape with many systems located in different data centers, with firewalls, without uniformised interfacing. Some data X is being replicated for availability and ...
0
votes
0answers
41 views

Synchronize Data From Multiple Data Sources

Our team is trying to build a predictive maintenance system whose task is to look at a set of events and predict whether these events depict a set of known anomalies or not. We are at the design ...
1
vote
2answers
240 views

How to discover discovery service

We are designing an architecture of a high availability distibuted system and we are considering server side vs client side service discovery. There is one problem, I need to solve - how is it ...
0
votes
1answer
313 views

Architecture for message processing with scheduling, at scale

I have to design the architecture of a system that processes messages in a distributed manner. If this were the only requirement, I would use a message queue like Kafka and distribute the work with ...
1
vote
1answer
95 views

Would this system be considered decentralized or distributed?

I am currently working on a report regarding a matrix multiplication distribution system I have been working on. But after reading a lot of somewhat contradictory sources online I have become in doubt ...
-1
votes
1answer
76 views

Documenting a distributed application [closed]

Recently, I've became the integration lead for a large distributed system. I have two main challenges before I can take on anything else: first document all system interfaces and it's relationships ...
0
votes
0answers
42 views

Do web companies use the master replica in data center nearest to the user to do writes?

I am interested how multi datacenter (multi homing in Google terms) systems handle writes to strongly consistent user data? Is the master replica for each user placed in the same datacenter and writes ...
1
vote
1answer
28 views

How does Operational Transformations work when offering offline editing capabilities?

As far as I understand, OT works really well with concurrent editing. There is only one remote copy of the document, and all editors update that remote copy simultaneously. There is a concept of lag ...
0
votes
1answer
96 views

Distribute jobs evenly among a set of servers

I have the need to distribute a set of long running jobs across a cluster of containers in ECS. These jobs essentially would need to open a socket connection with a remote server and begin streaming ...
0
votes
1answer
197 views

Robust locking mechanism between n service instances

I have a job that runs every hour. There can be n identical Node.js microservices running in a stack. The job must be executed only once per hour, though, so I've implemented the following locking ...
1
vote
0answers
50 views

Develop distributed system contain application with poor architecture [duplicate]

In last months the business continues dynamic grow so we must develop new features along distributed system. Unfortunately we do not have much time to deliver new solutions. The application is ...
1
vote
2answers
149 views

Designing a big data web app

How do you design a website that allows users to query a large amount of user data, more specifically: there are ~100 million users with ~100TB of data, data is stored in HDFS (not a database) number ...
4
votes
2answers
908 views

How to prevent overlap booking on a calendar booking system

As an exercise, I am trying to design a simple calendar booking system for multiple meeting rooms. I kind of got my head around some requirements such as find available rooms for a given time range, ...
1
vote
1answer
241 views

Configuration in a Distributed Application

I am developing a distributed application built as a collection of separate services which are served from multiple load balanced instances and trying to wrap my head around the distributed ...
1
vote
2answers
430 views

Message Broker in Client-Server-Applications

I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ). A simplified diagram of my ...
1
vote
1answer
188 views

Regarding data modelling in a NoSQL use case

I am reading the following link to learn about system designs of various systems. https://www.educative.io/collection/page/5668639101419520/5649050225344512/5673385510043648 (This is a paid link, I am ...
3
votes
2answers
239 views

Architecting a distributed file processing system with leadership election

I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice: Problem : There are a large number of files continuously posted on ...
1
vote
0answers
42 views

Byzantine Failures - Client Voter and Replica on Same Processor

I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial Dependent-Failures Output ...
0
votes
1answer
49 views

How to ensure that every log event will be delivered to the GrayLog

In our applications we traditionally log events locally into the logging files. As our applications are distributed on multiple server instances, searching for particular events are complicated and ...
1
vote
1answer
83 views

Processing data in a cluster with multiple tasks

The goal is to provide a scaleable system implementing multiple data processing tasks which can be seen as a graph. Data objects will travel that graph. Most object will visit the same nodes in the ...
1
vote
1answer
64 views

Publish / Subscribe Paradigm Implementation Problems

Reading the final chapter 6 of the paper "The Many Faces of Publish/Subscribe" I stumbled across: Decoupling is a desirable property because it enforces scalability at the abstraction level, by ...
-3
votes
1answer
42 views

Design of a Distributed System to Uniquely Assign Identifiers to Each Node

Let's say I want to design a distributed system whose only purpose is to assign each running node a unique identifier in some small range (Say, 5-100 nodes into a 10 bit identifier). When a node is ...
3
votes
0answers
270 views

Autosuggest at scale - trie sharding

While reading on the design for autosuggest implementation on large scale systems (like google), I'm able to understand the usage of trie and how top "n" terms are stored at each node to quickly ...
-1
votes
1answer
50 views

Whats the pros and cons of RPCs vs. HTTP in information exchange in microservice architectures?

What type of components could benefit from RPC over HTTP, vice-versa? Does there exist other information exchange mechanisms, what are their use? I've seen smart people use both, I'm sure for ...
0
votes
1answer
468 views

RabbitMQ: how to achieve scalable one-to-one communication?

I'm trying to use RabbitMQ for two-way one-to-one communication between clients and servers. So, Client 1 should be able to communicate only with Server 1, and vice versa: The servers stay always ...
0
votes
4answers
521 views

How to guarantee HTTP message delivery in fault tolerant way

When an application A communicates with an external/3rd party system B, there is always a chance that B is down. Say that A raises an event that should be sent as a message to B via HTTP, what is the ...
2
votes
3answers
92 views

Does a replicated distributed file system minimise the need for durability?

I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard. I'm also familiar with various key/value store type systems, some of which do not perform any system ...