Questions tagged [distributed-system]

This tag can be used by questions on distributed system concept, design, and implementations.

Filter by
Sorted by
Tagged with
1 vote
1 answer
377 views

Regarding data modeling in a NoSQL use case

I am reading the following link to learn about system designs of various systems. (This is a paid link, I am attaching all explanations below.) In an attempt to explain the system design of Instagram ...
user avatar
3 votes
2 answers
524 views

Architecting a distributed file processing system with leadership election

I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice: Problem : There are a large number of files continuously posted on ...
user avatar
1 vote
0 answers
50 views

Byzantine Failures - Client Voter and Replica on Same Processor

I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial Dependent-Failures Output ...
user avatar
  • 117
0 votes
1 answer
108 views

How to ensure that every log event will be delivered to the GrayLog

In our applications we traditionally log events locally into the logging files. As our applications are distributed on multiple server instances, searching for particular events are complicated and ...
user avatar
  • 9
1 vote
1 answer
294 views

Processing data in a cluster with multiple tasks

The goal is to provide a scaleable system implementing multiple data processing tasks which can be seen as a graph. Data objects will travel that graph. Most object will visit the same nodes in the ...
user avatar
  • 237
1 vote
1 answer
72 views

Publish / Subscribe Paradigm Implementation Problems

Reading the final chapter 6 of the paper "The Many Faces of Publish/Subscribe" I stumbled across: Decoupling is a desirable property because it enforces scalability at the abstraction level,...
user avatar
  • 119
-3 votes
1 answer
134 views

Design of a Distributed System to Uniquely Assign Identifiers to Each Node

Let's say I want to design a distributed system whose only purpose is to assign each running node a unique identifier in some small range (Say, 5-100 nodes into a 10 bit identifier). When a node is ...
user avatar
6 votes
1 answer
763 views

Autosuggest at scale - trie sharding

While reading on the design for autosuggest implementation on large scale systems (like google), I'm able to understand the usage of trie and how top "n" terms are stored at each node to quickly ...
user avatar
-1 votes
1 answer
51 views

Whats the pros and cons of RPCs vs. HTTP in information exchange in microservice architectures?

What type of components could benefit from RPC over HTTP, vice-versa? Does there exist other information exchange mechanisms, what are their use? I've seen smart people use both, I'm sure for ...
user avatar
1 vote
1 answer
1k views

RabbitMQ: how to achieve scalable one-to-one communication?

I'm trying to use RabbitMQ for two-way one-to-one communication between clients and servers. So, Client 1 should be able to communicate only with Server 1, and vice versa: The servers stay always ...
user avatar
  • 161
0 votes
4 answers
1k views

How to guarantee HTTP message delivery in fault tolerant way

When an application A communicates with an external/3rd party system B, there is always a chance that B is down. Say that A raises an event that should be sent as a message to B via HTTP, what is the ...
user avatar
  • 726
2 votes
3 answers
103 views

Does a replicated distributed file system minimise the need for durability?

I've been investigating various distributed file systems, like Gluster, Ceph, Moose and Lizard. I'm also familiar with various key/value store type systems, some of which do not perform any system ...
user avatar
  • 614
2 votes
2 answers
66 views

How to implement semantics of serializability in eventually consistent system?

In a distributed asynchronous system serializability cannot be achieved, but we still have to somehow provide the semantics. Consider quite natural requirement that user must have unique email or no ...
user avatar
4 votes
1 answer
185 views

Is the actor model suited for an application dealing with elapsed time?

I need to create a distributed application with timers / periodically checking two timestamps for elapsed time. Is it possible to achieve following behaviour with the actor model: Spawn Child actor ...
user avatar
  • 49
2 votes
0 answers
169 views

Testing an algorithm for safety in mutual exclusion

I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
user avatar
  • 181
3 votes
0 answers
53 views

Release Phase in Lamport's Mutual Exclusion Algorithm

I was studying Lamport's mutual exclusion algorithm from the original paper, and noticed a difference in the release phase to the one given in Wikipedia. In the original paper, author states that : ...
user avatar
  • 181
1 vote
2 answers
142 views

Strict definition of a Distributed System

I am taking an introductory course on Distributed Systems, and while reading a introductory tutorial the author discusses that 5 Rails servers behind a single load balancer all connected to one ...
user avatar
  • 181
2 votes
1 answer
259 views

System design for multiple workers

I need to design a system where there are many micro-services which we can think of as crawlers that poll 3rd party resource and if they find some new information they pass it to some other service ...
user avatar
-1 votes
1 answer
68 views

Writing a distributed chat logger

I'm trying to write a very high scale IRC logger that can handle like 1k messages per second. My Database of choice is cassandra and maybe later I might look into scylladb. My Question is how do I ...
user avatar
  • 101
1 vote
1 answer
65 views

Fail over for application that does outgoing requests

We built an application that has a list of fixed intranet URLs and crawls these periodically, transforms the data, stores it in a database. We have the requirement that the application has some kind ...
user avatar
  • 337
1 vote
0 answers
510 views

In a distributed system, how to prevent duplicated services from acting multiple times on a common data source?

We are currently rewriting our infrastructure from a monolith to a distributed system because the old system didn`t fit the workload anymore. In one part of our distributed system, we have an Entity (...
user avatar
  • 191
0 votes
1 answer
449 views

Validate Entity Key from another microservice comming from UI

Let's say an Orders service knows only about the CustomerId from Customer service. When the user places an order, the CustomerId gets transferred from the composite UI along with the OredrDto. The ...
user avatar
  • 209
1 vote
1 answer
225 views

Architecture for communication between multiple backend systems and IoT devices via WebSockets

I have a scenario in which thousands of IoT devices -- each fitted with a sim card -- are geographically distributed and shall be controlled individually. Therefore each IoT device has a WebSocket ...
user avatar
1 vote
1 answer
86 views

What does 'data coverage' mean when talking about ETL processes?

I was watching this talk about ETL's shortcomings and the solutions provided by the Kafka platform but I don't quite understand what the speaker is referring to when she says ETL tools have been ...
user avatar
1 vote
1 answer
261 views

How to architect master/slave store/messages and de-duplication?

To clarify, as I'm not entirely sure how to summarise in the title. I have a redis cluster and a redis-backed scheduler (similar to sidekiq/resque, etc). My redis cluster has a master and a couple ...
user avatar
5 votes
3 answers
6k views

Best practices for Heartbeat in distributed systems

We had in our system in the past an external data provider (call it source) sending regular heartbeats to a java application (call it client). If the heartbeat failed, system shut itself down (to ...
user avatar
  • 638
1 vote
1 answer
116 views

Only one application in cluster performs a job

I have a question about best practices given the following scenario. Requirements There is a daily process that has two steps: Query configuration data across dozens of database shards Perform ...
user avatar
1 vote
2 answers
81 views

what tool or methodology to use to do impact analisys in a distributed system?

We are working on a large (legacy) distributed system with various applications, messaging (soap services, database dblinks, rest services)... We are lacking for documentation and a big view of this ...
user avatar
  • 221
1 vote
1 answer
480 views

How to write integration tests for a distributed system highly dependant on current date and flow of time?

My team and I are trying to set up some integration tests of a distributed system that is highly dependant on the flow of time and events coming from external sources. Just to give you an idea about ...
user avatar
  • 241
4 votes
3 answers
348 views

Distributed database algorithm

I have a need for a distributed data store, where existing solutions may not work as the computers these will be running on will be extremely resource limited, for instance 64-128MB RAM. Plus, as a ...
user avatar
  • 149
12 votes
2 answers
7k views

How to safely run database migrations with multiple app instances?

We have an application that has a mix of both fast (< 1 second) and slow database migrations (> 30 seconds). Right now, we're running database migrations as a part of CI, but then our CI tool has ...
user avatar
  • 313
0 votes
0 answers
113 views

How to implement a fair queue, for multitanent, limitation per machine and limitation cross cluster?

I'm maintaining an already built system. The current system operates as following: Current situation: Client (UI) submits a request. Web layer round-robins to one of backends. Backend takes request ...
user avatar
  • 507
2 votes
1 answer
246 views

Design lease mechanism of shared resource

I have a shared resource in a database that is used by clients C1 and C2. Before the resource is read by C1, C1 should mark the resource as leased so that C2 will know it is busy. C1 will do some ...
user avatar
  • 415
4 votes
2 answers
89 views

Performance considerations with access control in a distributed system architecture

We're currently working on upgrading a small information system for an enterprise. Currently, the system has a corporate network zone (CORP) where services for internal users are placed, and a data ...
user avatar
  • 181
1 vote
1 answer
125 views

Distributed BiPartite Graph [closed]

I have a use case where I have a Bi-Partite Graph. Call one type of nodes "Type A" and the other "Type B." Now when a Type A node gets added, it forms some edges based on a criterion with Type B nodes ...
user avatar
  • 139
-3 votes
2 answers
156 views

Is the system with no central storage more secure?

Consider a distributed system S that consists of several processing nodes (N1, N2, N3, ...) and spans the geographical boundaries. S -> {N1, N2, N3, N4 ...} N is more than 1000. now consider that ...
user avatar
1 vote
2 answers
1k views

Store file in filesystem, and its metadata to the database atomicly

I have to store many pdf/jpg/png file of max 10mb in a filesystem, and need to save their metadata on a database. The SFTP and the DB may be on different nodes. On WS, I've a local db where I can ...
user avatar
-1 votes
2 answers
197 views

Bandwidth Congestion in a Large Distributed System

I've been working on a large distributed system typically with over 10,000 clients living on a single network (LAN) all accessing a central set of API's in the cloud. Not exactly a challenging problem....
user avatar
  • 429
4 votes
1 answer
834 views

distributed application design - using message broker

Background I'm trying to design a layer/component in my distributed application that will communicate between component A and component B. Right now, this "communication" is accomplished by ...
user avatar
  • 479
3 votes
0 answers
66 views

Sharing tasks among parallel independent jobs

I am getting files on HDFS and I have a legacy java app which checks for new files every 15 mins, does some transformations and write them to some remote server. Now, I am facing scaling issue with ...
user avatar
  • 209
0 votes
1 answer
195 views

Achieve atomicity between two separate systems

I have a lot of cases where I need to persist some information into a database and complete a secondary action using third party system. For example I have a trivial case, to save user information ...
user avatar
  • 745
2 votes
2 answers
496 views

What are the ideal use cases for weakly consistent systems?

I have been reading about distributed databases and the like providing weak and strong consistency guarantees, but perhaps I am short on imagination - when would you want to use a weakly consistent ...
user avatar
4 votes
3 answers
234 views

Consistency of diamonds in a concurrent system

I am struggling to find solutions to a problem that I encounter in a concurrent system (I am currently using the Actor Model). I basically have a diamond structure of actors: So D sends messages to C ...
user avatar
  • 149
1 vote
0 answers
102 views

Scaling Systems With Shared Data

I'm working on an in memory on disk cache and am starting to think about strategies for scaling the system. At the moment the system persists data by creating a file for each key value pair, where the ...
user avatar
  • 121
0 votes
1 answer
237 views

Why the distributed process clocks in a distributed system using Lamport clocks will eventually converge?

Basically the lamport clock helps the host to order the events rather than the exact time. How does all the logical clocks end up being converged in the long term?
user avatar
2 votes
2 answers
964 views

Decrement counter with high concurrency in distributed system

I am working on problem where a call to reduce a counter will come to a service and if counter is greater zero then call should be able to reduce it else fail. Pretty straightforward? huh! For a ...
user avatar
-1 votes
1 answer
237 views

How can a multiplayer game manage state over a local network? [closed]

I would like to create a multiplayer mobile game where players play together over an ad-hoc network, with no internet connection required. Players should be able to join and leave mid-game, and the ...
user avatar
  • 207
4 votes
1 answer
2k views

How does Redis (or any typical distributed cache) handle replication conflicts?

Suppose you set up a Redis cluster with one master and two slaves. Two clients are connected to each of the slaves. Both clients make conflicting changes at the same time: What happens if these ...
user avatar
  • 5,375
10 votes
3 answers
7k views

How acceptable is to keep business logic outside entities (in separate service classes)?

We were taught that objects are self contained things with data and behaviour and therefore they should have methods that act on their attributes. But there are several situations when this coupling ...
user avatar
  • 1,009
4 votes
1 answer
149 views

How are application in separate data centers routed

A bit of down time at work let my mind wonder a bit. I was designing a single location application on Digital Ocean (load balancer to app servers to DB with DNS). All fine. Then I thought what if the ...
user avatar
  • 945