Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [distributed-computing]

The tag has no usage guidance.

0
votes
1answer
35 views

Architecture patterns or SaaS solutions for throttling outgoing messages?

I'm implementing my service on AWS as a set of stateless EC2 nodes controlled by an ASG, backed by an RDS instance. I need to throttle the overall number of outgoing messages my service sends but can'...
2
votes
2answers
52 views

Architecting a distributed file processing system with leadership election

I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice: Problem : There are a large number of files continuously posted on ...
1
vote
0answers
37 views

Byzantine Failures - Client Voter and Replica on Same Processor

I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine Approach: A Tutorial Dependent-Failures Output ...
5
votes
2answers
178 views

How can we keep sight of business flows in event driven architectures?

I am planning to set up an event driven architecture using Spring Boot apps that publish and read messages from a Kafka broker. Let's suppose it were an e-commerce application with the usual events (...
3
votes
1answer
92 views

Is the actor model suited for an application dealing with elapsed time?

I need to create a distributed application with timers / periodically checking two timestamps for elapsed time. Is it possible to achieve following behaviour with the actor model: Spawn Child actor ...
2
votes
0answers
54 views

Testing an algorithm for safety in mutual exclusion

I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
1
vote
2answers
117 views

Distributed training of many small ML models

I have a data science application that involves training tens of thousands of small individual Gaussian models. By "small", I mean that any individual model can easily be trained on one of our worker ...
1
vote
2answers
94 views

How to avoid clashes in distributed computing when polling for work to do?

A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
-3
votes
1answer
186 views

How to deal with errors in a distributed system

In my experience, most errors in an application are bugs: Syntax Error Range Error (number out of bounds) Reference Error (undefined variable used) Type Error Custom Assertions These should be ...
2
votes
1answer
89 views

Docker and GPU-based computations. Feasible? [closed]

Recently I ran against this question and this Nvidia-docker project, which is an Nvidia Docker implementation and it made me wondering where, why and how this scheme makes sense? I found out some ...
1
vote
0answers
59 views

Optimistic locking vs separate lock store for background workers

I need to compute the value of a row R on a table based on rows queried on another table B. I need to do it in a queue worker because the computation of the value can be slow. A message is queued ...
1
vote
1answer
54 views

Only one application in cluster performs a job

I have a question about best practices given the following scenario. Requirements There is a daily process that has two steps: Query configuration data across dozens of database shards Perform ...
4
votes
3answers
300 views

Distributed database algorithm

I have a need for a distributed data store, where existing solutions may not work as the computers these will be running on will be extremely resource limited, for instance 64-128MB RAM. Plus, as a ...
0
votes
0answers
44 views

How to implement a fair queue, for multitanent, limitation per machine and limitation cross cluster?

I'm maintaining an already built system. The current system operates as following: Current situation: Client (UI) submits a request. Web layer round-robins to one of backends. Backend takes request ...
1
vote
1answer
75 views

Distributed BiPartite Graph [closed]

I have a use case where I have a Bi-Partite Graph. Call one type of nodes "Type A" and the other "Type B." Now when a Type A node gets added, it forms some edges based on a criterion with Type B nodes ...
0
votes
1answer
90 views

Removing Redis Entries from Multiple Redis Instances

I have an interesting problem that I'd like to solve. Let's say I have 2 deployments, US Data Center and Europe Data Center each containing Web Servers and a Redis Instance (replicated, but could ...
0
votes
1answer
727 views

WCF: Send events between services

I recently worked in a larger project where I touched the topic of distributed systems in the c# world the first time. I found that wcf is fantastic for remote procedure calls, but how do you add ...
4
votes
2answers
103 views

How to to send a file and record that it has been sent in an atomic way?

I'm not sure how to best approach this problem. I've got an object in my local database, for example this one: { title: "my-object", uploaded: false } I convert this to a file, say my-object.txt and ...
2
votes
0answers
58 views

Sharing tasks among parallel independent jobs

I am getting files on HDFS and I have a legacy java app which checks for new files every 15 mins, does some transformations and write them to some remote server. Now, I am facing scaling issue with ...
10
votes
3answers
408 views

How to handle side effects in CRQS when replaying events?

It is said that in CQRS it is easy to fix a bug, you just redeploy and then replay the events. But, what if one of the events should cause an external system not in your control to "ship an item" to ...
1
vote
2answers
133 views

Is caching data between independent but interacting systems okay?

We have two products let call them X and Y. Product-X is established and has numerous clients over continents whereas Product-Y's development was started a few years ago and it has yet to enjoy any ...
0
votes
1answer
90 views

Achieve atomicity between two separate systems

I have a lot of cases where I need to persist some information into a database and complete a secondary action using third party system. For example I have a trivial case, to save user information ...
-1
votes
1answer
33 views

Is there a name for a process object being serialized to move to a worker node?

There's a pattern for RPC or distributed computation I've seen a couple times and I'd like to know if there is a standard name for it so that I can communicate it more effectively. I'm describing it ...
4
votes
3answers
284 views

How to create simulator for distributed algorithms written in simple language

I started development of simulator for simulation of distributed algorithms in language C. My work consist of creating simple language for algorithm description and simulator which takes the described ...
2
votes
3answers
389 views

Data consistency in distributed systems

I wanted to ask this question that has been bugging my head for a long time. Recently, I have started to develop a distributed system which has continuous and frequent database lookups in a loop. Let ...
28
votes
4answers
2k views

How does a company like Amazon avoid bottlenecks accessing the database layer?

If you imagine a company like Amazon (or any other large e-commerce web application), that is operating an online store at massive scale and only has limited quantity of physical items in its ...
0
votes
2answers
550 views

What is total order and how can generate and maintain total order among the proposer proposals in paxos?

1) What is total order? Does total order mean the consecutive numbers must be strictly less than "<" or it can less than or equal to "<="? 2) how can generate and maintain total order among the ...
0
votes
0answers
46 views

Synchronized computation task distribution

I have a system where a set of Operations needs to be done every day. Each Operation has the following metadata: Type, Entity and a Date (independent of current date). It reads something from the ...
4
votes
2answers
127 views

load balancing in shared nothing cluster

I have a system where the load is distributed across the nodes in the cluster and once an entity is assigned to particular node, it is taken to completion through that node only(distribution is saved ...
6
votes
2answers
143 views

Does decentralization infer distribution?

I'm exploring the relationship between decentralization and distribution, and I've come to a question for which I've found no real explanation. Does the qualification of a system as "decentralized" ...
1
vote
1answer
148 views

#Apache-flink: Stream processing or Batch processing using Flink

I am tasked with redesigning an existing catalog processor and the requirement goes as below Requirement I have 5 to 10 vendors(each vendor can have multiple stores) who would provide me with 'XML' ...
3
votes
2answers
412 views

User interface and process that involves asynchronous communication between several microservices

System consists of several micro-services and each of them is in charge of it's own context, such as booking, payment, products and notifications. Let's imagine we have a traveler who is seeking ...
0
votes
2answers
549 views

Did I beat the CAP Theorem with this master-slaves distributed system (with picture)?

I was watching this video about the CAP theorem, where the author explains well the trade-offs of distributed systems. However I disagree with the CAP theorem in the following aspect. Given the ...
2
votes
2answers
113 views

SRP in the “big data” setting

We have a codebase at work that: Ingests (low) thousands of small files. Each of these input files contains about 50k “micro-items” These “micro-items” are then clustered together to find “macro-...
1
vote
1answer
82 views

Distributing an application in 2 or more servers

This is probably a common question and I've been reading a lot about this, but I can't look into a solution. Let's say I have an enterprise application in one glassfish server located in New York. ...
-2
votes
1answer
454 views

How do I implement autocomplete functionality for a big website? [closed]

I am looking to implement a custom autocomplete functionality for a travel website with huge amounts of data. I am thinking of implementing it using a trie. But how do I handle it if the data cannot ...
2
votes
0answers
118 views

How do I take data offline for a specific tenant, if their data is spread across multiple databases and multiple microservices?

I'm working on an application with about 50 or so microservices that have separate multi-tenanted databases. The problem arose of taking a tenant's data offline meaning that any requests for a certain ...
13
votes
2answers
2k views

Patterns for maintaining consistency in a distributed, event sourced system?

I've been reading on event sourcing lately and really like the ideas behind it but am stuck with the following problem. Let's say you have N concurrent processes which receive commands (e.g. web ...
5
votes
1answer
137 views

Distributed Calculation of Geometric Distance between vectors

I am looking at a large scale low latency way to calculate the geometric distance between vectors. Let's say I have a vector A. It has size 128 and of type 32 bit float. I want to get it's geometric ...
6
votes
3answers
1k views

What does a load balancer return?

When a user hits the load balancer and the load balancer determines which web server to forward to, what happens next? Does the load balancer forward the request and all its data to the webserver, ...
-2
votes
1answer
120 views

Investigating in a highly distributed environment [closed]

I was asked this by an interviewer today and unfortunately he wouldn't share the kind of answer he was looking for so I'm hoping maybe the community can help. Say you're an engineer at company X and ...
1
vote
0answers
34 views

Split long running work amongst multiple workers

I have a large number of rows in a database each containing a user specific token. For each token I need to maintain an open websocket connection. To scale this up I would like to run multiple ...
1
vote
2answers
839 views

Suggestions to distributed service pattern

We expose some webservices on a server, written in C#. Now we have to distribute the services to multiple servers, so the solution is distributed. And each request is dispatched to one the servers to ...
3
votes
1answer
333 views

Distributing work between TCP Clients

I have an application (run by a Windows Service) which connects to a TCP server (let's call the service 'Listener' from here on). [It's important to note that the TCP server is out of my reach, and ...
2
votes
1answer
2k views

Scaling websocket client connections (not server) to multiple servers

I wrote a Slack bot which must connect to Slack teams through websocket connections. Since the bot might be used by thousands of team, I will eventually need to distribute the teams across multiple ...
8
votes
2answers
598 views

Error handling in distributed system

This is the common sequence of two distributed components in our Java application: 1 A sends request to B 2 B starts some job J in parallel thread 3 B returns response to A 4 A accepts ...
3
votes
1answer
252 views

Is TCP still considered siginificant overhead compared to UDP? [closed]

I'm working on a distributed application for a graduate class I'm taking. There has been much discussion around implementing a way to maintain peer connection health statuses. Think Gossip. One of the ...
4
votes
3answers
198 views

How to store (and find) updated values in a Distributed Hash Table?

Distributed Hash Tables are a great way to store immutable data in a resilient way. However, I am wondering about how it would be possible to store new versions of old data in this system. Because ...
5
votes
1answer
2k views

Distributed long-running jobs

Scenario Various types of files containing customer's data are being uploaded via FTP, and a POST is being made to a REST API to update a database table, which tells the system (a web app) which of ...
3
votes
1answer
87 views

Coordinate access to resource between workers

I have a database accessed by an several servers, let's call them workers. These workers take one item from the database, do some operations on it and then update it back into the database. The ...