Questions tagged [distributed-computing]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
6
votes
1answer
195 views

How to prevent concurrency problems when using the repository pattern?

Considering that: when using the repository pattern you deal with entities - those are the atomic units you persist (regardless of Hibernate or "manually") when changing an entity you save it as a ...
-4
votes
1answer
77 views

What difference and relation are between fault tolerance and (high) availability?

From Coulouris' Distributed Systems 5ed Chapter 18 Replication 18.1 Introduction Increased availability: Users require services to be highly available. That is, the proportion ...
0
votes
0answers
43 views

Effective data/rows queuing out of database

A component in a system (called Notifier) inserts hundreds of messages ~300 (SMSs) per second in a table in MSSQL database with PENDING status by flagging a column called status with int value 0. ...
-3
votes
2answers
143 views

What is the difference between masking and tolerating failures?

Distributed Systems 5ed by Coulouris says on p21-22 1.5.5 Failure handling Detecting failures: Some failures can be detected. For example, checksums can be used to detect corrupted data in a ...
-2
votes
1answer
77 views

In the distributed computing problem known as the “Byzantine fault”, what differences and relations are between commander, lieutenant, and general? [closed]

Distributed systems by Coulouris says The Byzantine generals problem • In the informal statement of the Byzantine generals problem [Lamport et al. 1982], three or more generals are to agree to ...
-2
votes
0answers
73 views

Does the leader election problem only apply to process replication/redundancy?

The leader election problem is said to be In distributed computing, leader election is the process of designating a single process as the organizer of some task distributed among several computers (...
0
votes
1answer
52 views

Does the producer-consumer problem appear in both shared memory and distributed memory architectures?

I saw some books on operating system concepts mention the producer-consumer problem in the context of synchronizing concurrent accesses to shared resources. All seem to be in shared memory ...
2
votes
1answer
66 views

Do persistent/transient communication and temporal decoupling/coupling mean the same?

In Distributed Systems by Tanenbaum, p67 says CHAPTER 2. ARCHITECTURES 2.1. ARCHITECTURAL STYLES Cabri et al. [2000] provide a taxonomy of coordination models that can be applied ...
-3
votes
1answer
69 views

How to understand two-tier and three-tier solutions for web applications?

From Distributed Systems by Coulouris: Tiered architectures are complementary to layering. Whereas layering deals with the vertical organization of services into layers of abstraction, ...
0
votes
3answers
58 views

Is shared disk architecture scaling up or scaling out?

In Design Data Intensive Applications, If all you need is to scale to higher load, the simplest approach is to buy a more powerful machine (sometimes called vertical scaling or scaling up). ...
1
vote
1answer
75 views

Designing frontend experience to support backend async CQRS operations

I'm developing a microservice with CQRS and Event Sourcing. When an event is saved to the event store, the service currently also saves the updated aggregate root as a JSON object in a separate table. ...
1
vote
2answers
119 views

Develop a distributed pointer in C++

I s it possible to develop a distributed memory pointer in C++ by using operator overload? The idea would be that you pass ip address and port to the overloaded pointer *, &, and the pointer ...
1
vote
0answers
17 views

Consistency and Availability in distributed hashing Key value store

During the last semester of my college, we were asked to read the (chord paper) and implement it without fault tolerance. We built it with the following feature: A new node can join. A client can ...
2
votes
0answers
40 views

Distributing tasks to several clients

I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements. I have 1-...
3
votes
0answers
336 views

Using distributed locks in microservice environment

I'm developing distributed system and trying to use best practices of microservice architecture. I was faced with a situation when I think I need something like distributed locks. Since I have not so ...
-1
votes
1answer
142 views

How do decentralised systems deal with all nodes being killed

In relation to DHT's (distributed hash tables), how do decentralised systems deal with all nodes disconnecting and thus having its routing meta data invalidated? i.e with the implementation of DHT's ...
1
vote
0answers
78 views

Is the concept of Rx debounce possible in a distributed environment?

The issue I am building a server-side system that handles streams of time-series events from multiple users and I'd like to perform an action after a certain quiet period has elapsed in the event ...
1
vote
1answer
94 views

Would this system be considered decentralized or distributed?

I am currently working on a report regarding a matrix multiplication distribution system I have been working on. But after reading a lot of somewhat contradictory sources online I have become in doubt ...
0
votes
1answer
67 views

How to design a high-scale, reliable, distributed and periodic task (cronjob) execution service?

Assume you are tasked with implementing a "cronjob as a service" where you are supposed to be able to run potentially millions of periodical tasks (to simplify, making HTTP requests to URLs) with ...
3
votes
1answer
387 views

DynamoDB - Event Store on AWS

I'm designing an Event Store on AWS and I chose DynamoDB because it seemed the best option. My design seems to be quite good, but I'm facing some issues that I can't solve. The Design Events Events ...
0
votes
1answer
84 views

Distribute jobs evenly among a set of servers

I have the need to distribute a set of long running jobs across a cluster of containers in ECS. These jobs essentially would need to open a socket connection with a remote server and begin streaming ...
-2
votes
1answer
380 views

How to serve the front-end in a Spring microservice architecture?

I want to create the following distributed system: Spring back-end microservices containing the domain logic, a UAA (authentication) service, a Eureka service registry/discovery, a Spring Cloud Config ...
3
votes
2answers
226 views

Architecting a distributed file processing system with leadership election

I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice: Problem : There are a large number of files continuously posted on ...
1
vote
0answers
42 views

Byzantine Failures - Client Voter and Replica on Same Processor

I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial Dependent-Failures Output ...
5
votes
2answers
307 views

How can we keep sight of business flows in event driven architectures?

I am planning to set up an event driven architecture using Spring Boot apps that publish and read messages from a Kafka broker. Let's suppose it were an e-commerce application with the usual events (...
3
votes
1answer
108 views

Is the actor model suited for an application dealing with elapsed time?

I need to create a distributed application with timers / periodically checking two timestamps for elapsed time. Is it possible to achieve following behaviour with the actor model: Spawn Child actor ...
2
votes
0answers
89 views

Testing an algorithm for safety in mutual exclusion

I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
1
vote
2answers
250 views

Distributed training of many small ML models

I have a data science application that involves training tens of thousands of small individual Gaussian models. By "small", I mean that any individual model can easily be trained on one of our worker ...
1
vote
2answers
126 views

How to avoid clashes in distributed computing when polling for work to do?

A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
-3
votes
1answer
693 views

How to deal with errors in a distributed system

In my experience, most errors in an application are bugs: Syntax Error Range Error (number out of bounds) Reference Error (undefined variable used) Type Error Custom Assertions These should be ...
2
votes
1answer
154 views

Docker and GPU-based computations. Feasible? [closed]

Recently I ran against this question and this Nvidia-docker project, which is an Nvidia Docker implementation and it made me wondering where, why and how this scheme makes sense? I found out some ...
1
vote
0answers
72 views

Optimistic locking vs separate lock store for background workers

I need to compute the value of a row R on a table based on rows queried on another table B. I need to do it in a queue worker because the computation of the value can be slow. A message is queued ...
1
vote
1answer
111 views

Only one application in cluster performs a job

I have a question about best practices given the following scenario. Requirements There is a daily process that has two steps: Query configuration data across dozens of database shards Perform ...
4
votes
3answers
310 views

Distributed database algorithm

I have a need for a distributed data store, where existing solutions may not work as the computers these will be running on will be extremely resource limited, for instance 64-128MB RAM. Plus, as a ...
0
votes
0answers
105 views

How to implement a fair queue, for multitanent, limitation per machine and limitation cross cluster?

I'm maintaining an already built system. The current system operates as following: Current situation: Client (UI) submits a request. Web layer round-robins to one of backends. Backend takes request ...
1
vote
1answer
105 views

Distributed BiPartite Graph [closed]

I have a use case where I have a Bi-Partite Graph. Call one type of nodes "Type A" and the other "Type B." Now when a Type A node gets added, it forms some edges based on a criterion with Type B nodes ...
0
votes
1answer
129 views

Removing Redis Entries from Multiple Redis Instances

I have an interesting problem that I'd like to solve. Let's say I have 2 deployments, US Data Center and Europe Data Center each containing Web Servers and a Redis Instance (replicated, but could ...
0
votes
1answer
957 views

WCF: Send events between services

I recently worked in a larger project where I touched the topic of distributed systems in the c# world the first time. I found that wcf is fantastic for remote procedure calls, but how do you add ...
4
votes
2answers
104 views

How to to send a file and record that it has been sent in an atomic way?

I'm not sure how to best approach this problem. I've got an object in my local database, for example this one: { title: "my-object", uploaded: false } I convert this to a file, say my-object.txt and ...
3
votes
0answers
65 views

Sharing tasks among parallel independent jobs

I am getting files on HDFS and I have a legacy java app which checks for new files every 15 mins, does some transformations and write them to some remote server. Now, I am facing scaling issue with ...
10
votes
3answers
590 views

How to handle side effects in CRQS when replaying events?

It is said that in CQRS it is easy to fix a bug, you just redeploy and then replay the events. But, what if one of the events should cause an external system not in your control to "ship an item" to ...
1
vote
2answers
136 views

Is caching data between independent but interacting systems okay?

We have two products let call them X and Y. Product-X is established and has numerous clients over continents whereas Product-Y's development was started a few years ago and it has yet to enjoy any ...
0
votes
1answer
115 views

Achieve atomicity between two separate systems

I have a lot of cases where I need to persist some information into a database and complete a secondary action using third party system. For example I have a trivial case, to save user information ...
-1
votes
1answer
33 views

Is there a name for a process object being serialized to move to a worker node?

There's a pattern for RPC or distributed computation I've seen a couple times and I'd like to know if there is a standard name for it so that I can communicate it more effectively. I'm describing it ...
3
votes
3answers
376 views

How to create simulator for distributed algorithms written in simple language

I started development of simulator for simulation of distributed algorithms in language C. My work consist of creating simple language for algorithm description and simulator which takes the described ...
2
votes
3answers
510 views

Data consistency in distributed systems

I wanted to ask this question that has been bugging my head for a long time. Recently, I have started to develop a distributed system which has continuous and frequent database lookups in a loop. Let ...
29
votes
4answers
2k views

How does a company like Amazon avoid bottlenecks accessing the database layer?

If you imagine a company like Amazon (or any other large e-commerce web application), that is operating an online store at massive scale and only has limited quantity of physical items in its ...
0
votes
2answers
745 views

What is total order and how can generate and maintain total order among the proposer proposals in paxos?

1) What is total order? Does total order mean the consecutive numbers must be strictly less than "<" or it can less than or equal to "<="? 2) how can generate and maintain total order among the ...
0
votes
0answers
47 views

Synchronized computation task distribution

I have a system where a set of Operations needs to be done every day. Each Operation has the following metadata: Type, Entity and a Date (independent of current date). It reads something from the ...
3
votes
2answers
138 views

load balancing in shared nothing cluster

I have a system where the load is distributed across the nodes in the cluster and once an entity is assigned to particular node, it is taken to completion through that node only(distribution is saved ...