N8N Workers die every few hours

[Foreignpay]

Bug Description

I have 3 webhooks running, 1 master and 10 vorkers. The 10 workers are on a separate server and connected to the master using Docker Swarm.
The server where the Workers are located has 16 cores and 16GB of RAM allocated. Containers have a limit of 1.5 GB of RAM, the parallelism value is standard, equal to 10.
1-2 times a day (or at night) there is an abnormal spike in resource consumption, a queue starts to appear, which can't go through (we have to delete it through the database). A screenshot from Grafana is attached.
After reloading all Workers - everything comes back to normal.
What can be done about it? This is a critical bug because of which the team is thinking of moving to Python.

To Reproduce

We don't know.

We don't notice overload on the server or on the database. We don't see any regularity, but it most often happens in the evening and at night.

The load from the outside does not increase in this case.in the database. It doesn't happen regularly

Expected behavior

I really want stability and no glitches like this.

Operating System

Ubuntu Linux 22.04

n8n Version

1.67.1 (we tried a lot of different versions since 1.60)

Node.js Version

Docker

Database

PostgreSQL

Execution mode

queue

[Joffcom]

Hey @Foreignpay,

We have created an internal ticket to look into this which we will be tracking as "GHC-463"

[Joffcom]

Hey @Foreignpay

Are you reporting a bug or trying to raise a support issue?

[Foreignpay]

A screenshot from Grafana is attached.

[Foreignpay]

Hey @Foreignpay

Are you reporting a bug or trying to raise a support issue?

It seems like a bug. We have not found any explanation for this, and only a reboot helps.

[netroy]

This sounds like you have an active workflow that's causing it.
Would it be possible for you to try disabling workflows in batches to narrow down the workflow causing this?

[Foreignpay]

Idk, because we provide acquiring services.
It's too hard to disable it.