Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [distributed-computing]

The tag has no usage guidance.

0
votes
0answers
48 views

Is the concept of Rx debounce possible in a distributed environment?

The issue I am building a server-side system that handles streams of time-series events from multiple users and I'd like to perform an action after a certain quiet period has elapsed in the event ...
1
vote
1answer
90 views

Would this system be considered decentralized or distributed?

I am currently working on a report regarding a matrix multiplication distribution system I have been working on. But after reading a lot of somewhat contradictory sources online I have become in doubt ...
0
votes
1answer
42 views

How to design a high-scale, reliable, distributed and periodic task (cronjob) execution service?

Assume you are tasked with implementing a "cronjob as a service" where you are supposed to be able to run potentially millions of periodical tasks (to simplify, making HTTP requests to URLs) with ...
1
vote
1answer
47 views

DynamoDB - Event Store on AWS

I'm designing an Event Store on AWS and I chose DynamoDB because it seemed the best option. My design seems to be quite good, but I'm facing some issues that I can't solve. The Design Events Events ...
0
votes
1answer
40 views

Distribute jobs evenly among a set of servers

I have the need to distribute a set of long running jobs across a cluster of containers in ECS. These jobs essentially would need to open a socket connection with a remote server and begin streaming ...
-2
votes
1answer
66 views

How to serve the front-end in a Spring microservice architecture?

I want to create the following distributed system: Spring back-end microservices containing the domain logic, a UAA (authentication) service, a Eureka service registry/discovery, a Spring Cloud Config ...
0
votes
1answer
52 views

Architecture patterns or SaaS solutions for throttling outgoing messages?

I'm implementing my service on AWS as a set of stateless EC2 nodes controlled by an ASG, backed by an RDS instance. I need to throttle the overall number of outgoing messages my service sends but can'...
2
votes
2answers
78 views

Architecting a distributed file processing system with leadership election

I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice: Problem : There are a large number of files continuously posted on ...
1
vote
0answers
38 views

Byzantine Failures - Client Voter and Replica on Same Processor

I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial Dependent-Failures Output ...
5
votes
2answers
212 views

How can we keep sight of business flows in event driven architectures?

I am planning to set up an event driven architecture using Spring Boot apps that publish and read messages from a Kafka broker. Let's suppose it were an e-commerce application with the usual events (...
3
votes
1answer
100 views

Is the actor model suited for an application dealing with elapsed time?

I need to create a distributed application with timers / periodically checking two timestamps for elapsed time. Is it possible to achieve following behaviour with the actor model: Spawn Child actor ...
2
votes
0answers
63 views

Testing an algorithm for safety in mutual exclusion

I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
1
vote
2answers
152 views

Distributed training of many small ML models

I have a data science application that involves training tens of thousands of small individual Gaussian models. By "small", I mean that any individual model can easily be trained on one of our worker ...
1
vote
2answers
98 views

How to avoid clashes in distributed computing when polling for work to do?

A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
-3
votes
1answer
353 views

How to deal with errors in a distributed system

In my experience, most errors in an application are bugs: Syntax Error Range Error (number out of bounds) Reference Error (undefined variable used) Type Error Custom Assertions These should be ...
2
votes
1answer
104 views

Docker and GPU-based computations. Feasible? [closed]

Recently I ran against this question and this Nvidia-docker project, which is an Nvidia Docker implementation and it made me wondering where, why and how this scheme makes sense? I found out some ...
1
vote
0answers
63 views

Optimistic locking vs separate lock store for background workers

I need to compute the value of a row R on a table based on rows queried on another table B. I need to do it in a queue worker because the computation of the value can be slow. A message is queued ...
1
vote
1answer
108 views

Only one application in cluster performs a job

I have a question about best practices given the following scenario. Requirements There is a daily process that has two steps: Query configuration data across dozens of database shards Perform ...
4
votes
3answers
303 views

Distributed database algorithm

I have a need for a distributed data store, where existing solutions may not work as the computers these will be running on will be extremely resource limited, for instance 64-128MB RAM. Plus, as a ...
0
votes
0answers
97 views

How to implement a fair queue, for multitanent, limitation per machine and limitation cross cluster?

I'm maintaining an already built system. The current system operates as following: Current situation: Client (UI) submits a request. Web layer round-robins to one of backends. Backend takes request ...
1
vote
1answer
93 views

Distributed BiPartite Graph [closed]

I have a use case where I have a Bi-Partite Graph. Call one type of nodes "Type A" and the other "Type B." Now when a Type A node gets added, it forms some edges based on a criterion with Type B nodes ...
0
votes
1answer
107 views

Removing Redis Entries from Multiple Redis Instances

I have an interesting problem that I'd like to solve. Let's say I have 2 deployments, US Data Center and Europe Data Center each containing Web Servers and a Redis Instance (replicated, but could ...
0
votes
1answer
838 views

WCF: Send events between services

I recently worked in a larger project where I touched the topic of distributed systems in the c# world the first time. I found that wcf is fantastic for remote procedure calls, but how do you add ...
4
votes
2answers
103 views

How to to send a file and record that it has been sent in an atomic way?

I'm not sure how to best approach this problem. I've got an object in my local database, for example this one: { title: "my-object", uploaded: false } I convert this to a file, say my-object.txt and ...
2
votes
0answers
62 views

Sharing tasks among parallel independent jobs

I am getting files on HDFS and I have a legacy java app which checks for new files every 15 mins, does some transformations and write them to some remote server. Now, I am facing scaling issue with ...
10
votes
3answers
461 views

How to handle side effects in CRQS when replaying events?

It is said that in CQRS it is easy to fix a bug, you just redeploy and then replay the events. But, what if one of the events should cause an external system not in your control to "ship an item" to ...
1
vote
2answers
134 views

Is caching data between independent but interacting systems okay?

We have two products let call them X and Y. Product-X is established and has numerous clients over continents whereas Product-Y's development was started a few years ago and it has yet to enjoy any ...
0
votes
1answer
97 views

Achieve atomicity between two separate systems

I have a lot of cases where I need to persist some information into a database and complete a secondary action using third party system. For example I have a trivial case, to save user information ...
-1
votes
1answer
33 views

Is there a name for a process object being serialized to move to a worker node?

There's a pattern for RPC or distributed computation I've seen a couple times and I'd like to know if there is a standard name for it so that I can communicate it more effectively. I'm describing it ...
4
votes
3answers
323 views

How to create simulator for distributed algorithms written in simple language

I started development of simulator for simulation of distributed algorithms in language C. My work consist of creating simple language for algorithm description and simulator which takes the described ...
2
votes
3answers
443 views

Data consistency in distributed systems

I wanted to ask this question that has been bugging my head for a long time. Recently, I have started to develop a distributed system which has continuous and frequent database lookups in a loop. Let ...
29
votes
4answers
2k views

How does a company like Amazon avoid bottlenecks accessing the database layer?

If you imagine a company like Amazon (or any other large e-commerce web application), that is operating an online store at massive scale and only has limited quantity of physical items in its ...
0
votes
2answers
651 views

What is total order and how can generate and maintain total order among the proposer proposals in paxos?

1) What is total order? Does total order mean the consecutive numbers must be strictly less than "<" or it can less than or equal to "<="? 2) how can generate and maintain total order among the ...
0
votes
0answers
46 views

Synchronized computation task distribution

I have a system where a set of Operations needs to be done every day. Each Operation has the following metadata: Type, Entity and a Date (independent of current date). It reads something from the ...
4
votes
2answers
132 views

load balancing in shared nothing cluster

I have a system where the load is distributed across the nodes in the cluster and once an entity is assigned to particular node, it is taken to completion through that node only(distribution is saved ...
6
votes
2answers
156 views

Does decentralization infer distribution?

I'm exploring the relationship between decentralization and distribution, and I've come to a question for which I've found no real explanation. Does the qualification of a system as "decentralized" ...
1
vote
1answer
159 views

#Apache-flink: Stream processing or Batch processing using Flink

I am tasked with redesigning an existing catalog processor and the requirement goes as below Requirement I have 5 to 10 vendors(each vendor can have multiple stores) who would provide me with 'XML' ...
3
votes
2answers
473 views

User interface and process that involves asynchronous communication between several microservices

System consists of several micro-services and each of them is in charge of it's own context, such as booking, payment, products and notifications. Let's imagine we have a traveler who is seeking ...
0
votes
2answers
641 views

Did I beat the CAP Theorem with this master-slaves distributed system (with picture)?

I was watching this video about the CAP theorem, where the author explains well the trade-offs of distributed systems. However I disagree with the CAP theorem in the following aspect. Given the ...
2
votes
2answers
114 views

SRP in the “big data” setting

We have a codebase at work that: Ingests (low) thousands of small files. Each of these input files contains about 50k “micro-items” These “micro-items” are then clustered together to find “macro-...
1
vote
1answer
82 views

Distributing an application in 2 or more servers

This is probably a common question and I've been reading a lot about this, but I can't look into a solution. Let's say I have an enterprise application in one glassfish server located in New York. ...
-2
votes
1answer
503 views

How do I implement autocomplete functionality for a big website? [closed]

I am looking to implement a custom autocomplete functionality for a travel website with huge amounts of data. I am thinking of implementing it using a trie. But how do I handle it if the data cannot ...
2
votes
0answers
125 views

How do I take data offline for a specific tenant, if their data is spread across multiple databases and multiple microservices?

I'm working on an application with about 50 or so microservices that have separate multi-tenanted databases. The problem arose of taking a tenant's data offline meaning that any requests for a certain ...
13
votes
2answers
2k views

Patterns for maintaining consistency in a distributed, event sourced system?

I've been reading on event sourcing lately and really like the ideas behind it but am stuck with the following problem. Let's say you have N concurrent processes which receive commands (e.g. web ...
5
votes
1answer
137 views

Distributed Calculation of Geometric Distance between vectors

I am looking at a large scale low latency way to calculate the geometric distance between vectors. Let's say I have a vector A. It has size 128 and of type 32 bit float. I want to get it's geometric ...
8
votes
3answers
2k views

What does a load balancer return?

When a user hits the load balancer and the load balancer determines which web server to forward to, what happens next? Does the load balancer forward the request and all its data to the webserver, ...
-2
votes
1answer
124 views

Investigating in a highly distributed environment [closed]

I was asked this by an interviewer today and unfortunately he wouldn't share the kind of answer he was looking for so I'm hoping maybe the community can help. Say you're an engineer at company X and ...
1
vote
0answers
35 views

Split long running work amongst multiple workers

I have a large number of rows in a database each containing a user specific token. For each token I need to maintain an open websocket connection. To scale this up I would like to run multiple ...
1
vote
2answers
916 views

Suggestions to distributed service pattern

We expose some webservices on a server, written in C#. Now we have to distribute the services to multiple servers, so the solution is distributed. And each request is dispatched to one the servers to ...
3
votes
1answer
366 views

Distributing work between TCP Clients

I have an application (run by a Windows Service) which connects to a TCP server (let's call the service 'Listener' from here on). [It's important to note that the TCP server is out of my reach, and ...