Excessive polling on `solid_queue_semaphores` causes CPU to spike to 100%

[salmonsteak1]

Hello,

May I clarify how concurrency controls work in Solid Queue? From reading the README on concurrency controls , I have a general idea that the dispatcher checks for blocked jobs in the solid_queue_blocked_execution table, and moves it to the solid_queue_ready_execution table if a lock can be acquired from the solid_queue_semaphores table, please correct me if I'm wrong

The reason I'm asking is because I recently experienced a CPU spike for my Postgres database and upon closer inspection, there were alot of queries like these:

UPDATE "solid_queue_semaphores" SET value = value + 1, expires_at = '2025-08-28 13:52:06.939437' WHERE "solid_queue_semaphores"."key" = 'testing_key_1' AND "solid_queue_semaphores"."value" < 30 /*job='TestJob'*/

Once I stopped enqueuing these jobs that had this key, the CPU spike dropped (and the queries were gone).

There were also a lot of these "transactional" and "tuple" lock that is probably on the solid_queue_semaphores table in the time the CPU spiked to 100%:

If what I read in the README was true, wouldn't that mean that only the dispatchers are hitting the solid_queue_semaphores table (which is a very small number at a polling rate of 1s)? I'm not sure why am I getting these huge spikes in locks and CPU

Thank you!

[rosa]

Hey @salmonsteak1! That query is from your jobs that are concurrency-controlled. They need to check if they can gain the semaphore to be allowed to proceed, and when they're done, they need to release the semaphore so that others can proceed. If the query is trying to increment the value (like the one you copied), it means they're releasing the semaphore. A decrement query would be used to gain the semaphore. Each concurrency controlled job will do these two queries, one before being enqueued and another one after completing execution.

[salmonsteak1]

Ah, I see. So would this mean that each solid queue worker would be querying the solid_queue_semaphores table when trying to take a job from the solid_queue_blocked_execution table (provided that there's none in the queue)? That would explain why I am seeing so many of such queries to the semaphores table.

I also noticed that the number of in-progress jobs is lower than expected, even though there’s a backlog of jobs in the queues and plenty of available workers. My suspicion is that many of these workers are stuck waiting on the semaphore table query I mentioned earlier. If this is true, is there anything we can do about it?

Thank you!

[rosa]

Is it possible that the jobs are blocked? So the workers aren't waiting on the queries, but rather on the jobs to become ready.

[salmonsteak1]

Yes, we would sometimes have > 100k blocked jobs. Does having blocked jobs means that we would have workers "busy waiting" for these jobs in the blocked executions table, rather than picking up jobs from the ready executions table?

Thank you!

[rosa]

No, workers only poll the ready jobs and don't wait on blocked jobs. If workers are idle, they'll pick jobs from ready. Workers don't poll the semaphores table in any case, they indirectly try to update their own semaphore when they finish running a job, but that's it.