Perform separate queries per queue assigned to worker #46
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rosa
force-pushed
the
single-query-per-queue
branch
from
75be855 to
2dee1a6
Compare
We do this in the order specified in the worker configuration, and by priority within each queue. It's still possible to specify "*", which means all queues, and in that case the order will be only by priority. Paused queues aren't queried. The way it works when there are more than one queue in the list is as follows. Imagine the worker is configured to run ``` queue1, queue2, queue3 ``` In that order. Then: - Query as most N (N is the worker's capacity, or its number of idle threads) jobs from queue1. If we got at least N jobs, stop and assign these to the pool, and continue querying jobs from queue1. - If we got fewer jobs than N, query jobs from queue2. If we reached N, stop and post those to the pool, and go back to queue1. - If we still have space because we haven't filled N, go on to queue3 and do the same. Then go back to queue1.
rosa
force-pushed
the
single-query-per-queue
branch
2 times, most recently
from
3b76e75 to
b8b489e
Compare
rosa
commented
Nov 22, 2023
| @@ -12,7 +12,7 @@ def claiming(job_ids, process_id, &block) | |||
| job_data = Array(job_ids).collect { |job_id| { job_id: job_id, process_id: process_id } } | |||
|
|
|||
| insert_all(job_data) | |||
| where(job_id: job_ids).tap do |claimed| | |||
| where(job_id: job_ids).load.tap do |claimed| | |||
There was a problem hiding this comment.
Calling load here to avoid extra queries when checking the size of the relation and then loading the relation itself.
rosa
force-pushed
the
single-query-per-queue
branch
2 times, most recently
from
01333c5 to
15cbf56
Compare
When all priorities are the same.
rosa
force-pushed
the
single-query-per-queue
branch
from
15cbf56 to
78ca4b8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We do this in the order specified in the worker configuration and by priority within each queue. It's still possible to specify
"*", which means all queues, and in that case, the order will be only by priority. Paused queues aren't queried.The way it works when there is more than one queue in the list is as follows. Imagine the worker is configured to run
In that order. Then:
queue1. If we got at least N jobs, stop and assign these to the pool and continue querying jobs fromqueue1.queue2. If we reach N, stop and post those to the pool, and go back toqueue1.queue3and do the same. Then go back toqueue1.