Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [distributed-system]

The tag has no usage guidance.

-1
votes
1answer
48 views

Documenting a distributed application [closed]

Recently, I've became the integration lead for a large distributed system. I have two main challenges before I can take on anything else: first document all system interfaces and it's relationships ...
0
votes
0answers
41 views

Do web companies use the master replica in data center nearest to the user to do writes?

I am interested how multi datacenter (multi homing in Google terms) systems handle writes to strongly consistent user data? Is the master replica for each user placed in the same datacenter and writes ...
1
vote
1answer
24 views

How does Operational Transformations work when offering offline editing capabilities?

As far as I understand, OT works really well with concurrent editing. There is only one remote copy of the document, and all editors update that remote copy simultaneously. There is a concept of lag ...
0
votes
1answer
37 views

Distribute jobs evenly among a set of servers

I have the need to distribute a set of long running jobs across a cluster of containers in ECS. These jobs essentially would need to open a socket connection with a remote server and begin streaming ...
0
votes
1answer
36 views

Robust locking mechanism between n service instances

I have a job that runs every hour. There can be n identical Node.js microservices running in a stack. The job must be executed only once per hour, though, so I've implemented the following locking ...
1
vote
0answers
46 views

Develop distributed system contain application with poor architecture [duplicate]

In last months the business continues dynamic grow so we must develop new features along distributed system. Unfortunately we do not have much time to deliver new solutions. The application is ...
1
vote
2answers
105 views

Designing a big data web app

How do you design a website that allows users to query a large amount of user data, more specifically: there are ~100 million users with ~100TB of data, data is stored in HDFS (not a database) number ...
3
votes
2answers
213 views

How to prevent overlap booking on a calendar booking system

As an exercise, I am trying to design a simple calendar booking system for multiple meeting rooms. I kind of got my head around some requirements such as find available rooms for a given time range, ...
0
votes
1answer
66 views

Configuration in a Distributed Application

I am developing a distributed application built as a collection of separate services which are served from multiple load balanced instances and trying to wrap my head around the distributed ...
1
vote
2answers
70 views

Message Broker in Client-Server-Applications

I'm currently planning an architecture for a system that should consist of multiple microservices. We definitely want to make use of a message broker (probably RabbitMQ). A simplified diagram of my ...
1
vote
1answer
53 views

Regarding data modelling in a NoSQL use case

I am reading the following link to learn about system designs of various systems. https://www.educative.io/collection/page/5668639101419520/5649050225344512/5673385510043648 (This is a paid link, I am ...
2
votes
2answers
69 views

Architecting a distributed file processing system with leadership election

I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice: Problem : There are a large number of files continuously posted on ...
1
vote
0answers
37 views

Byzantine Failures - Client Voter and Replica on Same Processor

I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial Dependent-Failures Output ...
0
votes
1answer
28 views

How to ensure that every log event will be delivered to the GrayLog

In our applications we traditionally log events locally into the logging files. As our applications are distributed on multiple server instances, searching for particular events are complicated and ...
0
votes
1answer
38 views

Processing data in a cluster with multiple tasks

The goal is to provide a scaleable system implementing multiple data processing tasks which can be seen as a graph. Data objects will travel that graph. Most object will visit the same nodes in the ...
1
vote
1answer
48 views

Publish / Subscribe Paradigm Implementation Problems

Reading the final chapter 6 of the paper "The Many Faces of Publish/Subscribe" I stumbled across: Decoupling is a desirable property because it enforces scalability at the abstraction level, by ...
-3
votes
1answer
36 views

Design of a Distributed System to Uniquely Assign Identifiers to Each Node

Let's say I want to design a distributed system whose only purpose is to assign each running node a unique identifier in some small range (Say, 5-100 nodes into a 10 bit identifier). When a node is ...
2
votes
0answers
105 views

Autosuggest at scale - trie sharding

While reading on the design for autosuggest implementation on large scale systems (like google), I'm able to understand the usage of trie and how top "n" terms are stored at each node to quickly ...
-1
votes
1answer
47 views

Whats the pros and cons of RPCs vs. HTTP in information exchange in microservice architectures?

What type of components could benefit from RPC over HTTP, vice-versa? Does there exist other information exchange mechanisms, what are their use? I've seen smart people use both, I'm sure for ...
0
votes
1answer
175 views

RabbitMQ: how to achieve scalable one-to-one communication?

I'm trying to use RabbitMQ for two-way one-to-one communication between clients and servers. So, Client 1 should be able to communicate only with Server 1, and vice versa: The servers stay always ...
0
votes
4answers
215 views

How to guarantee HTTP message delivery in fault tolerant way

When an application A communicates with an external/3rd party system B, there is always a chance that B is down. Say that A raises an event that should be sent as a message to B via HTTP, what is the ...
0
votes
1answer
57 views

Does a replicated distributed file system minimise the need for durability?

I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard. I'm also familiar with various key/value store type systems, some of which do not perform any system ...
2
votes
2answers
57 views

How to implement semantics of serializability in eventually consistent system?

In a distributed asynchronous system serializability cannot be achieved, but we still have to somehow provide the semantics. Consider quite natural requirement that user must have unique email or no ...
3
votes
1answer
96 views

Is the actor model suited for an application dealing with elapsed time?

I need to create a distributed application with timers / periodically checking two timestamps for elapsed time. Is it possible to achieve following behaviour with the actor model: Spawn Child actor ...
2
votes
0answers
59 views

Testing an algorithm for safety in mutual exclusion

I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
3
votes
0answers
46 views

Release Phase in Lamport's Mutual Exclusion Algorithm

I was studying Lamport's mutual exclusion algorithm from the original paper, and noticed a difference in the release phase to the one given in Wikipedia. In the original paper, author states that : ...
0
votes
2answers
95 views

Strict definition of a Distributed System

I am taking an introductory course on Distributed Systems, and while reading a introductory tutorial the author discusses that 5 Rails servers behind a single load balancer all connected to one ...
2
votes
1answer
59 views

System design for multiple workers

I need to design a system where there are many micro-services which we can think of as crawlers that poll 3rd party resource and if they find some new information they pass it to some other service ...
-1
votes
1answer
64 views

Writing a distributed chat logger

I'm trying to write a very high scale IRC logger that can handle like 1k messages per second. My Database of choice is cassandra and maybe later I might look into scylladb. My Question is how do I ...
1
vote
1answer
52 views

Fail over for application that does outgoing requests

We built an application that has a list of fixed intranet URLs and crawls these periodically, transforms the data, stores it in a database. We have the requirement that the application has some kind ...
1
vote
0answers
65 views

In a distributed system, how to prevent duplicated services from acting multiple times on a common data source?

We are currently rewriting our infrastructure from a monolith to a distributed system because the old system didn`t fit the workload anymore. In one part of our distributed system, we have an Entity (...
0
votes
1answer
26 views

Validate Entity Key from another microservice comming from UI

Let's say an Orders service knows only about the CustomerId from Customer service. When the user places an order, the CustomerId gets transferred from the composite UI along with the OredrDto. The ...
1
vote
1answer
79 views

Architecture for communication between multiple backend systems and IoT devices via WebSockets

I have a scenario in which thousands of IoT devices -- each fitted with a sim card -- are geographically distributed and shall be controlled individually. Therefore each IoT device has a WebSocket ...
-1
votes
0answers
33 views

Service Discovery and services with multiple protocols

I have a use case where I want to ask a service discovery node about available services. All of these services will have a REST API, but some will also have a Vert.x event bus. The REST endpoints will ...
1
vote
1answer
48 views

What does 'data coverage' mean when talking about ETL processes?

I was watching this talk about ETL's shortcomings and the solutions provided by the Kafka platform but I don't quite understand what the speaker is referring to when she says ETL tools have been ...
0
votes
0answers
49 views

Exp backoff per endpoint or globally?

I have a system that can talk to any of a set of endpoints (pops). Should I keep track of an exponential backoff for reconnects per endpoint, or globally? The endpoints all talk to the same backend, ...
1
vote
1answer
115 views

How to architect master/slave store/messages and de-duplication?

To clarify, as I'm not entirely sure how to summarise in the title. I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc). My redis cluster has a master and a couple ...
5
votes
3answers
2k views

Best practices for Heartbeat in distributed systems

We had in our system in the past an external data provider (call it source) sending regular heartbeats to a java application (call it client). If the heartbeat failed, system shut itself down (to ...
1
vote
1answer
80 views

Only one application in cluster performs a job

I have a question about best practices given the following scenario. Requirements There is a daily process that has two steps: Query configuration data across dozens of database shards Perform ...
1
vote
2answers
73 views

what tool or methodology to use to do impact analisys in a distributed system?

We are working on a large (legacy) distributed system with various applications, messaging (soap services, database dblinks, rest services)... We are lacking for documentation and a big view of this ...
1
vote
1answer
262 views

How to write integration tests for a distributed system highly dependant on current date and flow of time?

My team and I are trying to set up some integration tests of a distributed system that is highly dependant on the flow of time and events coming from external sources. Just to give you an idea about ...
4
votes
3answers
300 views

Distributed database algorithm

I have a need for a distributed data store, where existing solutions may not work as the computers these will be running on will be extremely resource limited, for instance 64-128MB RAM. Plus, as a ...
8
votes
2answers
2k views

How to safely run database migrations with multiple app instances?

We have an application that has a mix of both fast (< 1 second) and slow database migrations (> 30 seconds). Right now, we're running database migrations as a part of CI, but then our CI tool has ...
0
votes
0answers
70 views

How to implement a fair queue, for multitanent, limitation per machine and limitation cross cluster?

I'm maintaining an already built system. The current system operates as following: Current situation: Client (UI) submits a request. Web layer round-robins to one of backends. Backend takes request ...
3
votes
1answer
140 views

Design lease mechanism of shared resource

I have a shared resource in a database that is used by clients C1 and C2. Before the resource is read by C1, C1 should mark the resource as leased so that C2 will know it is busy. C1 will do some ...
5
votes
2answers
70 views

Performance considerations with access control in a distributed system architecture

We're currently working on upgrading a small information system for an enterprise. Currently, the system has a corporate network zone (CORP) where services for internal users are placed, and a data ...
1
vote
1answer
89 views

Distributed BiPartite Graph [closed]

I have a use case where I have a Bi-Partite Graph. Call one type of nodes "Type A" and the other "Type B." Now when a Type A node gets added, it forms some edges based on a criterion with Type B nodes ...
-3
votes
2answers
145 views

Is the system with no central storage more secure?

Consider a distributed system S that consists of several processing nodes (N1, N2, N3, ...) and spans the geographical boundaries. S -> {N1, N2, N3, N4 ...} N is more than 1000. now consider that ...
1
vote
0answers
89 views

Cache sync across nodes in a cluster

I recently started working on a Spring web application, in which a cache for some heavily used data is stored in the JVM of each node in the AWS cluster. Any change in any of the node values results ...
1
vote
2answers
348 views

Store file in filesystem, and its metadata to the database atomicly

I have to store many pdf/jpg/png file of max 10mb in a filesystem, and need to save their metadata on a database. The SFTP and the DB may be on different nodes. On WS, I've a local db where I can ...