Questions tagged [distributed-computing]
Distributed computing refers to architectures where software components of a same system run on several processes distributed across a network of computers and somehow coordinated.
160 questions
0
votes
1
answer
85
views
How to rebalance data across nodes?
I am implementing a message queue where messages are distributed across nodes in a cluster. The goal is to design a system to be able to auto-scale without needing to keep a global map of each message ...
- 115
1
vote
2
answers
181
views
Generating unique number in distributed environment
I have little app that i plan to run behind load balancer, with up to 5-8 instances.
Inside the app has service, that needs to generate unique number that has up to 10 digits ( so 1-10 digits) each ...
- 227
-1
votes
3
answers
462
views
Avoid race condition using RDBMS transaction
I am working on design of a distributed system which will process some events. For simplicity lets say, multiple instance of same service will be consuming from same queue, every message will have a ...
3
votes
4
answers
883
views
Giving multiple components access to a single database
This is in context of a distributed computing. There is a Service A which owns a database and hosts APIs for updating an entities in this database.
As time went by, the service has evolved and we are ...
- 41
3
votes
1
answer
927
views
Long-running compute-intensive tasks in APIs: background workers?
I'm working on a design for an HTTP based API that takes in requests to perform a long-running task that requires CPU and RAM intensive processing.
To give an impression of the compute requirements, ...
- 139
0
votes
1
answer
64
views
What are the approaches for joining data in distributed processing
Let's suppose I have a processing workflow of:
Where:
The workflow can have many instances running at the same time.
A1 -> B1, C1 -> D1
A2 -> B2, C2 -> D2
and so on...
Each node is an ...
- 119
0
votes
1
answer
88
views
How to load consistent test data into multiple applications (which communicate by message streams)?
I have a system landscape consisting of multiple applications.
Each application provides REST interfaces to communicate with the outside world.
Each application has its own data storage (some kind of ...
- 207
-1
votes
1
answer
125
views
Pinning Stateful Tasks in Stateless microservice cluster
We have a microservice whose multiple instances are running for High Availablity and Scaling. But although we have multipe stateless microservice there are some tasks which needs to be singleton.
For ...
- 97
0
votes
1
answer
470
views
Large file uploading in chunks
How to upload large(mb/gb) of video files from client to app server?
The solution as of now I know is pretty simple and widely used is to break file in chunks at at client side and send http post ...
- 81
2
votes
2
answers
491
views
Is sequential consistency equivalent to performing memory accesses by a processes in program order and performing each memory access atomically?
In Fundamentals of Parallel Multicore Architecture, by Yan Solihin, p304 defines sequential consistency memory model:
Overall, we can express programmers’ implicit expectation of memory
access ...
- 5,495
4
votes
2
answers
161
views
Do coherence and transactional isolation mean the same?
About coherence vs consistency, https://en.wikipedia.org/wiki/Consistency_model says
Coherence deals with maintaining a global order in which writes to a
single location or single variable are seen ...
- 5,495
3
votes
1
answer
194
views
Database consistency for a distributed system
I'm working on a phone call monitoring project. The aim is to have one row in the calls table for each ongoing call. Each call instance may be updated by many different servers/threads as more ...
- 141
6
votes
3
answers
318
views
How can adding redunancy adversely affect performance
I have been reading about system design and was going through this popular article - A Word on Scalability
The author here says:
An always-on service is said to be scalable if adding resources to
...
- 171
1
vote
1
answer
81
views
Ordering of analytical events
My team is working on an analytics system for the web app. The web app has a service-oriented architecture, and all events are gathered in one centralized analytics service. The system is sending ...
- 77
2
votes
1
answer
103
views
Manager node makes HTTP request to self?
I am working on a distributed application that uses the Manager Worker pattern. The application's manager node submits work to worker nodes through a WorkerNode object (allowing WorkerNode to be ...
- 139
0
votes
1
answer
110
views
Implementing the microservice pattern
We are developing a system that runs a certain kind of simulation for our customers:
Every simulation runs on it's own server.
We typically have a two digit number of simulations running.
A client ...
- 223
3
votes
2
answers
848
views
Multiple sources of truth - Optimistic concurrency & Eventual consistency
I have a couple of web applications that write to their own databases. They also share a few entities, for example, the customer entity. My business case is such that the same field for the same ...
- 39
0
votes
2
answers
243
views
Cancelling a background service from another microservice in load balanced environment
I have 2 microservices [A], and [B]. These microservices run in a load balanced environment and there may be N instances if each microservice running. The microservices communicate via a message ...
- 491
2
votes
2
answers
3k
views
How do atomic updates work at scale?
I am working on a weird ID system where I generate all the IDs in advance and need to atomically pop() one from the database, so no ID is read twice at the same time (and so used more than once). This ...
- 2,661
0
votes
1
answer
396
views
what is the difference between "distributed data processing" and "distributed computing"?
While going through learning hadoop and spark, I came across "distributed data processing" and "distributed computing".
Could you let me know if they both are same or referring to different concepts?
- 123
7
votes
1
answer
2k
views
How to prevent concurrency problems when using the repository pattern?
Considering that:
when using the repository pattern you deal with entities - those are the atomic units you persist (regardless of Hibernate or "manually")
when changing an entity you save it as a ...
- 271
-4
votes
1
answer
634
views
What difference and relation are between fault tolerance and (high) availability?
From Coulouris' Distributed Systems 5ed
Chapter 18 Replication
18.1 Introduction
Increased availability: Users require services to be highly available. That is, the proportion ...
- 5,495
0
votes
0
answers
51
views
Effective data/rows queuing out of database
A component in a system (called Notifier) inserts hundreds of messages ~300 (SMSs) per second in a table in MSSQL database with PENDING status by flagging a column called status with int value 0.
...
- 109
-3
votes
2
answers
1k
views
What is the difference between masking and tolerating failures?
Distributed Systems 5ed by Coulouris says on p21-22
1.5.5 Failure handling
Detecting failures: Some failures can be detected. For example, checksums can be used to detect corrupted data in a ...
- 5,495
-2
votes
1
answer
118
views
In the distributed computing problem known as the "Byzantine fault", what differences and relations are between commander, lieutenant, and general? [closed]
Distributed systems by Coulouris says
The Byzantine generals problem •
In the informal statement of the Byzantine generals problem [Lamport et al. 1982], three or more generals are to agree to ...
- 5,495
1
vote
1
answer
828
views
Does the producer-consumer problem appear in both shared memory and distributed memory architectures?
I saw some books on operating system concepts mention the producer-consumer problem in the context of synchronizing concurrent accesses to shared resources. All seem to be in shared memory ...
- 5,495
2
votes
1
answer
576
views
Do persistent/transient communication and temporal decoupling/coupling mean the same?
In Distributed Systems by Tanenbaum, p67 says
CHAPTER 2. ARCHITECTURES
2.1. ARCHITECTURAL STYLES
Cabri et al. [2000] provide a taxonomy of coordination models that can
be applied ...
- 5,495
-2
votes
1
answer
733
views
How to understand two-tier and three-tier solutions for web applications?
From Distributed Systems by Coulouris:
Tiered architectures are complementary to layering. Whereas
layering deals with the vertical organization of services into layers
of abstraction, tiering ...
- 5,495
1
vote
3
answers
316
views
Is shared disk architecture scaling up or scaling out?
In Design Data Intensive Applications,
If all you need is to scale to higher load, the simplest approach is to buy a more powerful machine (sometimes called vertical scaling or scaling up). ...
- 5,495
1
vote
1
answer
272
views
Designing frontend experience to support backend async CQRS operations
I'm developing a microservice with CQRS and Event Sourcing. When an event is saved to the event store, the service currently also saves the updated aggregate root as a JSON object in a separate table. ...
- 127
1
vote
2
answers
219
views
Develop a distributed pointer in C++
I s it possible to develop a distributed memory pointer in C++ by using operator overload?
The idea would be that you pass ip address and port to the overloaded pointer *, &, and the pointer ...
0
votes
1
answer
233
views
Consistency and Availability in distributed hashing Key value store
During the last semester of my college, we were asked to read the (chord paper) and implement it without fault tolerance. We built it with the following feature:
A new node can join.
A client can ...
- 135
2
votes
0
answers
164
views
Distributing tasks to several clients
I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements.
I have 1-...
- 121
4
votes
2
answers
2k
views
Using distributed locks in microservice environment
I'm developing distributed system and trying to use best practices of microservice architecture.
I was faced with a situation when I think I need something like distributed locks. Since I have not so ...
-1
votes
1
answer
162
views
How do decentralised systems deal with all nodes being killed
In relation to DHT's (distributed hash tables), how do decentralised systems deal with all nodes disconnecting and thus having its routing meta data invalidated? i.e with the implementation of DHT's ...
- 107
3
votes
0
answers
499
views
Is the concept of Rx debounce possible in a distributed environment?
The issue
I am building a server-side system that handles streams of time-series events from multiple users and I'd like to perform an action after a certain quiet period has elapsed in the event ...
- 139
1
vote
1
answer
129
views
Would this system be considered decentralized or distributed?
I am currently working on a report regarding a matrix multiplication distribution system I have been working on. But after reading a lot of somewhat contradictory sources online I have become in doubt ...
- 19
1
vote
1
answer
816
views
How to design a high-scale, reliable, distributed and periodic task (cronjob) execution service?
Assume you are tasked with implementing a "cronjob as a service" where you are supposed to be able to run potentially millions of periodical tasks (to simplify, making HTTP requests to URLs) with ...
- 113
3
votes
1
answer
851
views
DynamoDB - Event Store on AWS
I'm designing an Event Store on AWS and I chose DynamoDB because it seemed the best option. My design seems to be quite good, but I'm facing some issues that I can't solve.
The Design
Events
Events ...
0
votes
1
answer
1k
views
Distribute jobs evenly among a set of servers
I have the need to distribute a set of long running jobs across a cluster of containers in ECS. These jobs essentially would need to open a socket connection with a remote server and begin streaming ...
- 9
-2
votes
1
answer
740
views
How to serve the front-end in a Spring microservice architecture?
I want to create the following distributed system: Spring back-end microservices containing the domain logic, a UAA (authentication) service, a Eureka service registry/discovery, a Spring Cloud Config ...
- 483
3
votes
2
answers
793
views
Architecting a distributed file processing system with leadership election
I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice:
Problem : There are a large number of files continuously posted on ...
- 139
1
vote
0
answers
56
views
Byzantine Failures - Client Voter and Replica on Same Processor
I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine
Approach: A Tutorial
Dependent-Failures Output ...
- 117
5
votes
2
answers
1k
views
How can we keep sight of business flows in event driven architectures?
I am planning to set up an event driven architecture using Spring Boot apps that publish and read messages from a Kafka broker.
Let's suppose it were an e-commerce application with the usual events (...
- 427
5
votes
1
answer
211
views
Is the actor model suited for an application dealing with elapsed time?
I need to create a distributed application with timers / periodically checking two timestamps for elapsed time.
Is it possible to achieve following behaviour with the actor model:
Spawn Child actor
...
- 59
2
votes
0
answers
193
views
Testing an algorithm for safety in mutual exclusion
I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
- 181
2
votes
2
answers
405
views
Distributed training of many small ML models
I have a data science application that involves training tens of thousands of small individual Gaussian models. By "small", I mean that any individual model can easily be trained on one of our worker ...
- 137
1
vote
2
answers
458
views
How to avoid clashes in distributed computing when polling for work to do?
A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
- 61
-3
votes
1
answer
1k
views
How to deal with errors in a distributed system
In my experience, most errors in an application are bugs:
Syntax Error
Range Error (number out of bounds)
Reference Error (undefined variable used)
Type Error
Custom Assertions
These should be ...
- 2,661
2
votes
1
answer
200
views
Docker and GPU-based computations. Feasible? [closed]
Recently I ran against this question and this Nvidia-docker project, which is an Nvidia Docker implementation and it made me wondering where, why and how this scheme makes sense?
I found out some ...
- 129