Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [distributed-system]

The tag has no usage guidance.

0
votes
4answers
120 views

How to guarantee HTTP message delivery in fault tolerant way

When an application A communicates with an external/3rd party system B, there is always a chance that B is down. Say that A raises an event that should be sent as a message to B via HTTP, what is the ...
0
votes
1answer
52 views

Does a replicated distributed file system minimise the need for durability?

I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard. I'm also familiar with various key/value store type systems, some of which do not perform any system ...
2
votes
2answers
55 views

How to implement semantics of serializability in eventually consistent system?

In a distributed asynchronous system serializability cannot be achieved, but we still have to somehow provide the semantics. Consider quite natural requirement that user must have unique email or no ...
3
votes
1answer
87 views

Is the actor model suited for an application dealing with elapsed time?

I need to create a distributed application with timers / periodically checking two timestamps for elapsed time. Is it possible to achieve following behaviour with the actor model: Spawn Child actor ...
2
votes
0answers
44 views

Testing an algorithm for safety in mutual exclusion

I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
3
votes
0answers
35 views

Release Phase in Lamport's Mutual Exclusion Algorithm

I was studying Lamport's mutual exclusion algorithm from the original paper, and noticed a difference in the release phase to the one given in Wikipedia. In the original paper, author states that : ...
0
votes
0answers
41 views

How distributed Elastic search works at high level?

I am wondering how product search(in scalable system) can be implemented using distributed Elastic search . Here are the high level data points i can think of :- Consider there are n shards . Shard ...
0
votes
2answers
85 views

Strict definition of a Distributed System

I am taking an introductory course on Distributed Systems, and while reading a introductory tutorial the author discusses that 5 Rails servers behind a single load balancer all connected to one ...
2
votes
1answer
50 views

System design for multiple workers

I need to design a system where there are many micro-services which we can think of as crawlers that poll 3rd party resource and if they find some new information they pass it to some other service ...
-1
votes
1answer
62 views

Writing a distributed chat logger

I'm trying to write a very high scale IRC logger that can handle like 1k messages per second. My Database of choice is cassandra and maybe later I might look into scylladb. My Question is how do I ...
1
vote
1answer
50 views

Fail over for application that does outgoing requests

We built an application that has a list of fixed intranet URLs and crawls these periodically, transforms the data, stores it in a database. We have the requirement that the application has some kind ...
1
vote
0answers
45 views

In a distributed system, how to prevent duplicated services from acting multiple times on a common data source?

We are currently rewriting our infrastructure from a monolith to a distributed system because the old system didn`t fit the workload anymore. In one part of our distributed system, we have an Entity (...
0
votes
1answer
16 views

Validate Entity Key from another microservice comming from UI

Let's say an Orders service knows only about the CustomerId from Customer service. When the user places an order, the CustomerId gets transferred from the composite UI along with the OredrDto. The ...
1
vote
1answer
50 views

Architecture for communication between multiple backend systems and IoT devices via WebSockets

I have a scenario in which thousands of IoT devices -- each fitted with a sim card -- are geographically distributed and shall be controlled individually. Therefore each IoT device has a WebSocket ...
0
votes
0answers
29 views

Service Discovery and services with multiple protocols

I have a use case where I want to ask a service discovery node about available services. All of these services will have a REST API, but some will also have a Vert.x event bus. The REST endpoints will ...
1
vote
1answer
47 views

What does 'data coverage' mean when talking about ETL processes?

I was watching this talk about ETL's shortcomings and the solutions provided by the Kafka platform but I don't quite understand what the speaker is referring to when she says ETL tools have been ...
0
votes
0answers
47 views

Exp backoff per endpoint or globally?

I have a system that can talk to any of a set of endpoints (pops). Should I keep track of an exponential backoff for reconnects per endpoint, or globally? The endpoints all talk to the same backend, ...
1
vote
1answer
41 views

How to architect master/slave store/messages and de-duplication?

To clarify, as I'm not entirely sure how to summarise in the title. I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc). My redis cluster has a master and a couple ...
5
votes
3answers
642 views

Best practices for Heartbeat in distributed systems

We had in our system in the past an external data provider (call it source) sending regular heartbeats to a java application (call it client). If the heartbeat failed, system shut itself down (to ...
0
votes
0answers
252 views

Best options to get centralized PHP/MySQL app working in a distributed (offline) manner?

I have a situation where we have an old PHP/mySql application (10 years old, no framework, about 10 tables) that is used on a few facilities (less than 10) in different parts of the world. The main ...
1
vote
1answer
53 views

Only one application in cluster performs a job

I have a question about best practices given the following scenario. Requirements There is a daily process that has two steps: Query configuration data across dozens of database shards Perform ...
1
vote
2answers
71 views

what tool or methodology to use to do impact analisys in a distributed system?

We are working on a large (legacy) distributed system with various applications, messaging (soap services, database dblinks, rest services)... We are lacking for documentation and a big view of this ...
1
vote
1answer
177 views

How to write integration tests for a distributed system highly dependant on current date and flow of time?

My team and I are trying to set up some integration tests of a distributed system that is highly dependant on the flow of time and events coming from external sources. Just to give you an idea about ...
3
votes
2answers
246 views

Distributed database algorithm

I have a need for a distributed data store, where existing solutions may not work as the computers these will be running on will be extremely resource limited, for instance 64-128MB RAM. Plus, as a ...
7
votes
2answers
1k views

How to safely run database migrations with multiple app instances?

We have an application that has a mix of both fast (< 1 second) and slow database migrations (> 30 seconds). Right now, we're running database migrations as a part of CI, but then our CI tool has ...
0
votes
0answers
40 views

How to implement a fair queue, for multitanent, limitation per machine and limitation cross cluster?

I'm maintaining an already built system. The current system operates as following: Current situation: Client (UI) submits a request. Web layer round-robins to one of backends. Backend takes request ...
3
votes
1answer
106 views

Design lease mechanism of shared resource

I have a shared resource in a database that is used by clients C1 and C2. Before the resource is read by C1, C1 should mark the resource as leased so that C2 will know it is busy. C1 will do some ...
3
votes
0answers
54 views

Performance considerations with access control in a distributed system architecture

We're currently working on upgrading a small information system for an enterprise. Currently, the system has a corporate network zone (CORP) where services for internal users are placed, and a data ...
1
vote
1answer
70 views

Distributed BiPartite Graph [closed]

I have a use case where I have a Bi-Partite Graph. Call one type of nodes "Type A" and the other "Type B." Now when a Type A node gets added, it forms some edges based on a criterion with Type B nodes ...
-3
votes
2answers
145 views

Is the system with no central storage more secure?

Consider a distributed system S that consists of several processing nodes (N1, N2, N3, ...) and spans the geographical boundaries. S -> {N1, N2, N3, N4 ...} N is more than 1000. now consider that ...
1
vote
0answers
55 views

Cache sync across nodes in a cluster

I recently started working on a Spring web application, in which a cache for some heavily used data is stored in the JVM of each node in the AWS cluster. Any change in any of the node values results ...
1
vote
2answers
249 views

Store file in filesystem, and its metadata to the database atomicly

I have to store many pdf/jpg/png file of max 10mb in a filesystem, and need to save their metadata on a database. The SFTP and the DB may be on different nodes. On WS, I've a local db where I can ...
0
votes
0answers
134 views

How to design an async mechanism for performing long-running tasks in a workflow

I'm designing a workflow engine (that runs as a service) for executing arbitrarily defined workflows consisting of arbitrarily defined tasks. There will be a queue. An instance of the service will ...
0
votes
0answers
129 views

distributed system: design pattern to register slave with master

I am developing a distributed system for network discovery, where the user starts a job via a master's REST API. The master then gives the same task to several slaves (in different network segments)...
-1
votes
2answers
136 views

Bandwidth Congestion in a Large Distributed System

I've been working on a large distributed system typically with over 10,000 clients living on a single network (LAN) all accessing a central set of API's in the cloud. Not exactly a challenging problem....
4
votes
1answer
414 views

distributed application design - using message broker

Background I'm trying to design a layer/component in my distributed application that will communicate between component A and component B. Right now, this "communication" is accomplished by ...
2
votes
0answers
58 views

Sharing tasks among parallel independent jobs

I am getting files on HDFS and I have a legacy java app which checks for new files every 15 mins, does some transformations and write them to some remote server. Now, I am facing scaling issue with ...
0
votes
1answer
84 views

Achieve atomicity between two separate systems

I have a lot of cases where I need to persist some information into a database and complete a secondary action using third party system. For example I have a trivial case, to save user information ...
2
votes
2answers
136 views

What are the ideal use cases for weakly consistent systems?

I have been reading about distributed databases and the like providing weak and strong consistency guarantees, but perhaps I am short on imagination - when would you want to use a weakly consistent ...
4
votes
3answers
220 views

Consistency of diamonds in a concurrent system

I am struggling to find solutions to a problem that I encounter in a concurrent system (I am currently using the Actor Model). I basically have a diamond structure of actors: So D sends messages to C ...
1
vote
0answers
275 views

Difference between servants, dispatchers, and skeletons in RMI

I'm studying the implementation of RMI and am a bit confused regarding the difference between servants, dispatchers, and skeletons. If I understand correctly: Servants are what actually carries out ...
1
vote
0answers
86 views

Scaling Systems With Shared Data

I'm working on an in memory on disk cache and am starting to think about strategies for scaling the system. At the moment the system persists data by creating a file for each key value pair, where the ...
0
votes
1answer
164 views

Why the distributed process clocks in a distributed system using Lamport clocks will eventually converge?

Basically the lamport clock helps the host to order the events rather than the exact time. How does all the logical clocks end up being converged in the long term?
1
vote
2answers
247 views

Decrement counter with high concurrency in distributed system

I am working on problem where a call to reduce a counter will come to a service and if counter is greater zero then call should be able to reduce it else fail. Pretty straightforward? huh! For a ...
-1
votes
1answer
140 views

How can a multiplayer game manage state over a local network? [closed]

I would like to create a multiplayer mobile game where players play together over an ad-hoc network, with no internet connection required. Players should be able to join and leave mid-game, and the ...
3
votes
1answer
763 views

How does Redis (or any typical distributed cache) handle replication conflicts?

Suppose you set up a Redis cluster with one master and two slaves. Two clients are connected to each of the slaves. Both clients make conflicting changes at the same time: What happens if these ...
6
votes
3answers
2k views

How acceptable is to keep business logic outside entities (in separate service classes)?

We were taught that objects are self contained things with data and behaviour and therefore they should have methods that act on their attributes. But there are several situations when this coupling ...
4
votes
1answer
136 views

How are application in separate data centers routed

A bit of down time at work let my mind wonder a bit. I was designing a single location application on Digital Ocean (load balancer to app servers to DB with DNS). All fine. Then I thought what if the ...
0
votes
0answers
85 views

communications architecture for cross-platform, distributed, local cloud environment

I'm working on a project which has an end goal of providing the user with a local cloud environment in which they can, in an efficient manner, store and retrieve various pieces of data (files and ...
4
votes
2answers
125 views

load balancing in shared nothing cluster

I have a system where the load is distributed across the nodes in the cluster and once an entity is assigned to particular node, it is taken to completion through that node only(distribution is saved ...