Questions tagged [queue]
The queue tag has no usage guidance.
69 questions
0
votes
1
answer
107
views
Matching supersets
In my application I have requests and items, each of which are associated with property tuples (key value pairs, where keys can be repeated) stored in a Postgres database.
The goal is to find a single ...
- 159
-1
votes
1
answer
320
views
Airflow DAG vs custom worker to process queue of videos
I need to process a queue of videos in a scalable way. Processing includes number of tasks that can depend on each other. Some tasks are computationally expensive some not (e.g. transcribe audio, ...
- 1
0
votes
2
answers
3k
views
How to terminate python queue and instruct all consumer-threads to finish their tasks?
I have a multi-threaded application.
There is 1 thread that produces a resource and puts it into a queue and many (not constant amount) consumer-threads that get the resources from the queue.
When ...
- 111
0
votes
1
answer
210
views
Are self-written queues and linked lists really worth it (better than built-in arrays)?
Let's say I need a queue for my project and the language I use does not have a built-in queue structure. So I decided to implement one myself.
So I create an object with elements positions as the ...
- 39
3
votes
4
answers
1k
views
Iterating through stacks and queues?
Are stacks and queues allowed to be iterated over, conceptually?
It “feels wrong” for some reason but I can't come up with a reason why it would be bad to see through the whole thing that's stacked/...
- 159
1
vote
1
answer
425
views
What kind of queue should I use for processing large volume of data?
I have a segment of customers which may range between 1 and 5000. For each customers, I need to do some processing by making some query to the database using the data from the customer. The query is ...
- 121
2
votes
2
answers
1k
views
Is Kafka a good idea for connecting a REST API with Worker which runs large computations per job in a queue?
We currently have a REST API service (let's call it as A) written in Python which triggers various heavy calculation jobs for Worker (W) written in Python as well. Both services are interconnected ...
- 83
0
votes
1
answer
1k
views
Scheduling asynchronous jobs with efficient round-robin algorithm using Redis
We are trying to redesign our microservice architecture-based application to support multiple tenants. We have a simple queue service that will be utilized by other services to queue any asynchronous ...
- 101
-1
votes
1
answer
36
views
Options to maintain queue of resources that will be uniquely returned
I need to maintain a queue of resources in a cache such that when multiple clients try to pop elements from the queue concurrently, each element would be uniquely returned to only one of the clients. ...
- 157
-1
votes
1
answer
135
views
Design approach to decouple DB write to reduce application latency
I want to use DynamoDB to save application operation data at the end of a session for auditing purposes without affecting the actual application latency. Because this date is only for auditing so we ...
0
votes
0
answers
30
views
Is there a way to add impending state changes to a queue on a node server, and is it necessary?
I'm writing an express/socket.io-powered game server for a web game. I have a central map of game state objects, each representing an ongoing match, like so:
// map of gameId -> game (primary ...
- 1,114
1
vote
1
answer
1k
views
Pattern to limit message queue processing based on external API throttling
I have a project where there's an external API which implements throttling. Roughly speaking, I'm allowed to perform N requests per minute. I also have a message queue (Apache Kafka) whose consumers ...
- 157
-2
votes
1
answer
110
views
What is the proper way to define a generic cron-job
Am going to handle a CRON job module which will be a generic module, on the job execution time shall I push the events to a queue so let the respective module will pick and do the respective operation....
6
votes
2
answers
748
views
Design question for handling large volumes of messages in multi-tenant queue
I have a system with two applications interacting via a message queue. Let's call them Producer and Consumer. Some key context is that this a multi-tenancy scenario.
Producer produces events based ...
- 169
0
votes
1
answer
102
views
Distributed processing of a large number of queues
I have a rather basic application hosted on Kubernetes, which connects to a Mongo database.
The app has a wallet feature. A user can put money in their wallet using real-world payments (e.g. via ...
- 407
2
votes
3
answers
3k
views
why and when is queues used in backend architectures?
I am implementing a backend with a typescript framework called nestjs and I was reading some documentation about their usage of queues. https://docs.nestjs.com/techniques/queues I have never used ...
- 189
1
vote
0
answers
145
views
Asynchronous communication with legacy system
Background
My task is to create new system (X) for realtime work tracking for internal use. You should think it like an app that you have opened on a separate monitor/ on phone all day and you are ...
- 11
0
votes
0
answers
47
views
Designing sane messaging paradigm for concurrent updates
We have a weird use-case that I need to support in my application and needed some opinions on how to design it.
At a basic level the application allows users to work on a "list" of "tweets" she cares ...
- 109
-1
votes
1
answer
2k
views
How can I represent with UML a process that involves queues?
I would like to use UML diagrams to show some processes I am designing and would like to implement.
The processes involve using a queue, and adding and taking out elements from it.
In one of this ...
- 297
1
vote
2
answers
317
views
Worker pool running tasks of the same kind serially
I would like to run tasks in parallel. At this time, I am using a very simple
worker pool using a single concurrent queue shared by all the threads.
Every task has a non unique "tag" (an integer in ...
2
votes
0
answers
221
views
Database vs Logs - Persistence approach evaluation
I have a node js app, in which I need to perform background jobs. My plan is to use a data structure which will contain all the jobs needs done. My design also suggests that this data structure needs ...
- 1,159
1
vote
1
answer
871
views
Queue management project - Recommendations
I would like to develop a Patient Queue management system with online booking.
In this case there are options to generate queue tokens for various departments. For example dept A will have tokens ...
- 121
1
vote
1
answer
141
views
Do I need a queue manager over the database generating execution requests?
I am developing a multi-user application where a user interacts with the UI and submits execution requests (ER). Each user can submit multiple ERs one after the other and multiple users may submit ERs ...
- 147
0
votes
2
answers
1k
views
Best design for a AMQP work queue
I have a system where a user uploads a file to import some users but it's necessary to do some validations and this take some time. I would like to create a work queue with RabbitMQ and the user will ...
- 1
0
votes
1
answer
106
views
Multithreading - 2 active running threads calling each other without blocking itself
I have an implemented UDP receiver which is continuously receiving DTN packets (UDP encapsulated) and performing some operations on them. This whole process is working on a single main thread. However ...
- 13
3
votes
3
answers
297
views
Dealing with data arriving at a different times
I have a load balanced service that needs two data inputs to correctly give a result.
One of these inputs comes from a queue, the other comes from user input.
Most of the time the data that comes ...
- 4,057
1
vote
1
answer
322
views
Serving tasks for multiple users from one queue in parallel way
I'm designing a solution in which users perform tasks based on task queue. The point is to create system, where user goes on specific URL and server serves him listing with for example: items to asses ...
- 121
-1
votes
2
answers
329
views
How do job queues return results to UI?
I am working on an app that will run queries and return results to UI. However, these queries can take a long time to finish so I should not wait for the result. As far as I know, people use task ...
- 189
-3
votes
2
answers
708
views
Why are multiple backends in this system? [closed]
I am trying to understand the architecture of the system described in this patent about aggregating and analyzing confidential data:
https://patents.justia.com/patent/20180089196.
The general ...
- 105
9
votes
2
answers
2k
views
Fair distributed task scheduling with RabbitMQ
Problem: A single client of our system can completely flood all available resources with a massive workload. You can consider we have only one queue, and anyone can schedule any amount of work in it. ...
- 199
2
votes
3
answers
3k
views
Pull important messages from DLQ queue and save them in a relational database, to be analyzed and sent back later. Is it a good idea?
My project has a integration with a external system. We need to send some important informations to this system. For this, we create a micro service to connect to this external system. This micro ...
- 2,476
0
votes
1
answer
69
views
Modified priority queue (with "disabled" elements)
I need something like a priority queue but with the following modification:
Sometimes I need to mark some elements in the queue as "disabled".
Disabled elements:
temporarily (until marked non-...
- 791
1
vote
2
answers
584
views
Are there any use cases for List when Deques and Arrays are available?
I've been thinking about this over the past few weeks, and I've come up with no good arguments. My perspective is from Java, but if anyone has any language-specific cases outside of this language, I'...
- 219
3
votes
1
answer
326
views
How to name two functions that could be named the same?
I have a node app with a queue for processing jobs in the background. I have a file that exports a function which when run, creates a job in my queue. In that file, I also have the handler for this ...
-3
votes
1
answer
88
views
Best Data Structure for ordered list of connected users
I have three attributes for a user: connectionId (e.g. "sd4-h4sdg-u4j-a2rr3"), userName, and placeInLine.
I can't think of a single data structure that will handle my scenario of only allowing a ...
- 159
2
votes
3
answers
549
views
Why would I use a queue to list all files and subfolders recursively
In Robert Sedgewick's Algorithms book there's this exercise:
A folder is a list of files and folders. Write a program that takes
the name of a folder as a command-line argument and prints out all ...
1
vote
2
answers
2k
views
Best way to design a billing system for a SaaS
i am working on a SaaS project that provides some services that people can call via API. My problem is how do I build the billing algorithm. I want to charge on every call to the service.
Now I'm ...
- 209
-3
votes
3
answers
761
views
Priority Queue and Set in Java or general
why do we need priority queue when a set always keeps its object sorted, I think we can always remove the first/last element of a set and it would work as priority queue because priority queue offers ...
- 1
0
votes
1
answer
337
views
The right way of redundant task queuing?
I'm currently working on refacting an existing project. Basicaly, I need to perform requests to remote services. Current solution is to write tasks to a database and then poll them using cron jobs. ...
- 103
-1
votes
1
answer
106
views
Moving XML data into DB [closed]
Is this a "overkill" design to store XML data in a relational database?
One point is that these web API REST services can later be used by other applications.
- 13
6
votes
3
answers
1k
views
Large doubly-linked-list (or other) storage on disk for queue system; options for how to store
I'm looking at building a message queuing library in Go, which will be used as part of a larger application. A doubly-linked-list seems like a sensible approach for an in-memory data structure, but ...
- 239
0
votes
1
answer
2k
views
Scheduling notifications reminders for users
I am trying to design the best way to send reminders to users for events they are registered for:
reminders should be sent 72, 48 and 24 hours before the event
reminders cannot be sent twice (so user ...
- 101
4
votes
4
answers
4k
views
Designing an efficient implementation of a random access queue based on a linkedlist
I'm currently working through Robert Sedgewicks "Algorithms in java" (3rd edition, german) on my own and am currently at one of the more complicated questions there. I think I may have the starting ...
- 163
2
votes
1
answer
362
views
How to detect end of queue in a parallelized web crawler?
Let's say I want to program a parallelized web crawler which has a shared FIFO queue (multi consumer/multi producer). The queue only contains URLs. How do I detect the end of the queue?
A worker ...
- 41
6
votes
0
answers
206
views
I need to write my own version of a bounded queue [closed]
I need to write my own version of a bounded queue. It must be thread safe. The consumption of this queue is based on a priority system, where a consumer makes a request for an object in the queue.
...
- 179
2
votes
2
answers
1k
views
Queue how to preserve requests?
I'm working with Laravel queues lately.
I've all set up and running fine.
What I see is that I perform a http request to my system, the controllers manage the request issuing a Model having request ...
- 513
6
votes
3
answers
1k
views
Slow throughput - still worth using a dedicated message queue?
Here are some requirements for a queue:
Every few days add ~100k tasks with various priorities
Workers will pull tasks at typically less than 10 / second
Tasks need to be completed by ~2 unique ...
- 163
-2
votes
1
answer
64
views
Name for object to re-order events received in arbitrary order?
I have a program which generates events. They are ordered with a Lamport clock and sent over the network. They may arrive in a different order. On the receiving end I have an object which re-orders ...
- 176
10
votes
2
answers
3k
views
How to best design a job queue with constraints?
Consider the following situation:
You have a program which creates numerous 'jobs' that need to be processed and places them into a queue.
You have other worker programs which grab the next 'job' in ...
- 101
1
vote
2
answers
3k
views
Ordering 'Tasks' for execution based on their dependencies
Let's say I have a set of Tasks that they have dependencies. These tasks are not in order, but the execution of tasks should be in the proper order. Each task has two properties: Before and After.
...
- 1,993