Questions tagged [distributed-computing]

Distributed computing refers to architectures where software components of a same system run on several processes distributed across a network of computers and somehow coordinated.

Filter by
Sorted by
Tagged with
-1 votes
1 answer
45 views

What are the approaches for joining data in distributed processing

Let's suppose I have a processing workflow of: Where: The workflow can have many instances running at the same time. A1 -> B1, C1 -> D1 A2 -> B2, C2 -> D2 and so on... Each node is an ...
user avatar
  • 109
0 votes
1 answer
34 views

How to load consistent test data into multiple applications (which communicate by message streams)?

I have a system landscape consisting of multiple applications. Each application provides REST interfaces to communicate with the outside world. Each application has its own data storage (some kind of ...
user avatar
-1 votes
1 answer
90 views

Pinning Stateful Tasks in Stateless microservice cluster

We have a microservice whose multiple instances are running for High Availablity and Scaling. But although we have multipe stateless microservice there are some tasks which needs to be singleton. For ...
user avatar
0 votes
1 answer
121 views

Large file uploading in chunks

How to upload large(mb/gb) of video files from client to app server? The solution as of now I know is pretty simple and widely used is to break file in chunks at at client side and send http post ...
user avatar
  • 81
2 votes
2 answers
113 views

Is sequential consistency equivalent to performing memory accesses by a processes in program order and performing each memory access atomically?

In Fundamentals of Parallel Multicore Architecture, by Yan Solihin, p304 defines sequential consistency memory model: Overall, we can express programmers’ implicit expectation of memory access ...
user avatar
  • 5,271
4 votes
2 answers
108 views

Do coherence and transactional isolation mean the same?

About coherence vs consistency, https://en.wikipedia.org/wiki/Consistency_model says Coherence deals with maintaining a global order in which writes to a single location or single variable are seen ...
user avatar
  • 5,271
3 votes
1 answer
148 views

Database consistency for a distributed system

I'm working on a phone call monitoring project. The aim is to have one row in the calls table for each ongoing call. Each call instance may be updated by many different servers/threads as more ...
user avatar
  • 141
6 votes
3 answers
288 views

How can adding redunancy adversely affect performance

I have been reading about system design and was going through this popular article - A Word on Scalability The author here says: An always-on service is said to be scalable if adding resources to ...
user avatar
  • 171
1 vote
1 answer
74 views

Ordering of analytical events

My team is working on an analytics system for the web app. The web app has a service-oriented architecture, and all events are gathered in one centralized analytics service. The system is sending ...
user avatar
2 votes
1 answer
57 views

Manager node makes HTTP request to self?

I am working on a distributed application that uses the Manager Worker pattern. The application's manager node submits work to worker nodes through a WorkerNode object (allowing WorkerNode to be ...
user avatar
0 votes
1 answer
98 views

Implementing the microservice pattern

We are developing a system that runs a certain kind of simulation for our customers: Every simulation runs on it's own server. We typically have a two digit number of simulations running. A client ...
user avatar
1 vote
2 answers
163 views

Multiple sources of truth - Optimistic concurrency & Eventual consistency

I have a couple of web applications that write to their own databases. They also share a few entities, for example, the customer entity. My business case is such that the same field for the same ...
user avatar
  • 19
0 votes
2 answers
103 views

Cancelling a background service from another microservice in load balanced environment

I have 2 microservices [A], and [B]. These microservices run in a load balanced environment and there may be N instances if each microservice running. The microservices communicate via a message ...
user avatar
  • 441
1 vote
2 answers
1k views

How do atomic updates work at scale?

I am working on a weird ID system where I generate all the IDs in advance and need to atomically pop() one from the database, so no ID is read twice at the same time (and so used more than once). This ...
user avatar
  • 2,412
0 votes
1 answer
122 views

what is the difference between "distributed data processing" and "distributed computing"?

While going through learning hadoop and spark, I came across "distributed data processing" and "distributed computing". Could you let me know if they both are same or referring to different concepts?
user avatar
7 votes
1 answer
806 views

How to prevent concurrency problems when using the repository pattern?

Considering that: when using the repository pattern you deal with entities - those are the atomic units you persist (regardless of Hibernate or "manually") when changing an entity you save it as a ...
user avatar
-4 votes
1 answer
405 views

What difference and relation are between fault tolerance and (high) availability?

From Coulouris' Distributed Systems 5ed Chapter 18 Replication 18.1 Introduction Increased availability: Users require services to be highly available. That is, the proportion ...
user avatar
  • 5,271
0 votes
0 answers
45 views

Effective data/rows queuing out of database

A component in a system (called Notifier) inserts hundreds of messages ~300 (SMSs) per second in a table in MSSQL database with PENDING status by flagging a column called status with int value 0. ...
user avatar
-3 votes
2 answers
542 views

What is the difference between masking and tolerating failures?

Distributed Systems 5ed by Coulouris says on p21-22 1.5.5 Failure handling Detecting failures: Some failures can be detected. For example, checksums can be used to detect corrupted data in a ...
user avatar
  • 5,271
-2 votes
1 answer
97 views

In the distributed computing problem known as the "Byzantine fault", what differences and relations are between commander, lieutenant, and general? [closed]

Distributed systems by Coulouris says The Byzantine generals problem • In the informal statement of the Byzantine generals problem [Lamport et al. 1982], three or more generals are to agree to ...
user avatar
  • 5,271
1 vote
1 answer
581 views

Does the producer-consumer problem appear in both shared memory and distributed memory architectures?

I saw some books on operating system concepts mention the producer-consumer problem in the context of synchronizing concurrent accesses to shared resources. All seem to be in shared memory ...
user avatar
  • 5,271
2 votes
1 answer
288 views

Do persistent/transient communication and temporal decoupling/coupling mean the same?

In Distributed Systems by Tanenbaum, p67 says CHAPTER 2. ARCHITECTURES 2.1. ARCHITECTURAL STYLES Cabri et al. [2000] provide a taxonomy of coordination models that can be applied ...
user avatar
  • 5,271
-2 votes
1 answer
470 views

How to understand two-tier and three-tier solutions for web applications?

From Distributed Systems by Coulouris: Tiered architectures are complementary to layering. Whereas layering deals with the vertical organization of services into layers of abstraction, tiering ...
user avatar
  • 5,271
1 vote
3 answers
211 views

Is shared disk architecture scaling up or scaling out?

In Design Data Intensive Applications, If all you need is to scale to higher load, the simplest approach is to buy a more powerful machine (sometimes called vertical scaling or scaling up). ...
user avatar
  • 5,271
1 vote
1 answer
132 views

Designing frontend experience to support backend async CQRS operations

I'm developing a microservice with CQRS and Event Sourcing. When an event is saved to the event store, the service currently also saves the updated aggregate root as a JSON object in a separate table. ...
user avatar
1 vote
2 answers
162 views

Develop a distributed pointer in C++

I s it possible to develop a distributed memory pointer in C++ by using operator overload? The idea would be that you pass ip address and port to the overloaded pointer *, &, and the pointer ...
user avatar
0 votes
1 answer
123 views

Consistency and Availability in distributed hashing Key value store

During the last semester of my college, we were asked to read the (chord paper) and implement it without fault tolerance. We built it with the following feature: A new node can join. A client can ...
user avatar
  • 135
2 votes
0 answers
69 views

Distributing tasks to several clients

I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements. I have 1-...
user avatar
3 votes
1 answer
1k views

Using distributed locks in microservice environment

I'm developing distributed system and trying to use best practices of microservice architecture. I was faced with a situation when I think I need something like distributed locks. Since I have not so ...
user avatar
-1 votes
1 answer
149 views

How do decentralised systems deal with all nodes being killed

In relation to DHT's (distributed hash tables), how do decentralised systems deal with all nodes disconnecting and thus having its routing meta data invalidated? i.e with the implementation of DHT's ...
user avatar
3 votes
0 answers
337 views

Is the concept of Rx debounce possible in a distributed environment?

The issue I am building a server-side system that handles streams of time-series events from multiple users and I'd like to perform an action after a certain quiet period has elapsed in the event ...
user avatar
  • 139
1 vote
1 answer
105 views

Would this system be considered decentralized or distributed?

I am currently working on a report regarding a matrix multiplication distribution system I have been working on. But after reading a lot of somewhat contradictory sources online I have become in doubt ...
user avatar
1 vote
1 answer
510 views

How to design a high-scale, reliable, distributed and periodic task (cronjob) execution service?

Assume you are tasked with implementing a "cronjob as a service" where you are supposed to be able to run potentially millions of periodical tasks (to simplify, making HTTP requests to URLs) with ...
user avatar
3 votes
1 answer
759 views

DynamoDB - Event Store on AWS

I'm designing an Event Store on AWS and I chose DynamoDB because it seemed the best option. My design seems to be quite good, but I'm facing some issues that I can't solve. The Design Events Events ...
user avatar
0 votes
1 answer
691 views

Distribute jobs evenly among a set of servers

I have the need to distribute a set of long running jobs across a cluster of containers in ECS. These jobs essentially would need to open a socket connection with a remote server and begin streaming ...
user avatar
-2 votes
1 answer
622 views

How to serve the front-end in a Spring microservice architecture?

I want to create the following distributed system: Spring back-end microservices containing the domain logic, a UAA (authentication) service, a Eureka service registry/discovery, a Spring Cloud Config ...
user avatar
3 votes
2 answers
500 views

Architecting a distributed file processing system with leadership election

I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice: Problem : There are a large number of files continuously posted on ...
user avatar
1 vote
0 answers
50 views

Byzantine Failures - Client Voter and Replica on Same Processor

I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial Dependent-Failures Output ...
user avatar
  • 117
5 votes
2 answers
755 views

How can we keep sight of business flows in event driven architectures?

I am planning to set up an event driven architecture using Spring Boot apps that publish and read messages from a Kafka broker. Let's suppose it were an e-commerce application with the usual events (...
user avatar
4 votes
1 answer
180 views

Is the actor model suited for an application dealing with elapsed time?

I need to create a distributed application with timers / periodically checking two timestamps for elapsed time. Is it possible to achieve following behaviour with the actor model: Spawn Child actor ...
user avatar
  • 49
2 votes
0 answers
169 views

Testing an algorithm for safety in mutual exclusion

I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
user avatar
  • 181
2 votes
2 answers
367 views

Distributed training of many small ML models

I have a data science application that involves training tens of thousands of small individual Gaussian models. By "small", I mean that any individual model can easily be trained on one of our worker ...
user avatar
  • 137
1 vote
2 answers
315 views

How to avoid clashes in distributed computing when polling for work to do?

A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
user avatar
-3 votes
1 answer
1k views

How to deal with errors in a distributed system

In my experience, most errors in an application are bugs: Syntax Error Range Error (number out of bounds) Reference Error (undefined variable used) Type Error Custom Assertions These should be ...
user avatar
  • 2,412
2 votes
1 answer
180 views

Docker and GPU-based computations. Feasible? [closed]

Recently I ran against this question and this Nvidia-docker project, which is an Nvidia Docker implementation and it made me wondering where, why and how this scheme makes sense? I found out some ...
user avatar
1 vote
0 answers
97 views

Optimistic locking vs separate lock store for background workers

I need to compute the value of a row R on a table based on rows queried on another table B. I need to do it in a queue worker because the computation of the value can be slow. A message is queued ...
user avatar
1 vote
1 answer
116 views

Only one application in cluster performs a job

I have a question about best practices given the following scenario. Requirements There is a daily process that has two steps: Query configuration data across dozens of database shards Perform ...
user avatar
4 votes
3 answers
340 views

Distributed database algorithm

I have a need for a distributed data store, where existing solutions may not work as the computers these will be running on will be extremely resource limited, for instance 64-128MB RAM. Plus, as a ...
user avatar
0 votes
0 answers
113 views

How to implement a fair queue, for multitanent, limitation per machine and limitation cross cluster?

I'm maintaining an already built system. The current system operates as following: Current situation: Client (UI) submits a request. Web layer round-robins to one of backends. Backend takes request ...
user avatar
  • 507
1 vote
1 answer
125 views

Distributed BiPartite Graph [closed]

I have a use case where I have a Bi-Partite Graph. Call one type of nodes "Type A" and the other "Type B." Now when a Type A node gets added, it forms some edges based on a criterion with Type B nodes ...
user avatar
  • 139