[Questions] How can I reduce ra_log_ets memory?

[KOCEAN33]

Community Support Policy

• I have read RabbitMQ's Community Support Policy
• I agree to provide all relevant information (versions, logs, rabbitmq-diagnostics output, detailed reproduction steps)

RabbitMQ version used

3.13.7

Erlang version used

26.2.x

Operating system (distribution) used

Kubernetes 1.29

How is RabbitMQ deployed?

Kubernetes Operator(s) from Team RabbitMQ

rabbitmq-diagnostics status output

See https://www.rabbitmq.com/docs/cli to learn how to use rabbitmq-diagnostics

Runtime

OS PID: 11
OS: Linux
Uptime (seconds): 73210
Is under maintenance?: false
RabbitMQ version: 3.13.7
RabbitMQ release series support status: see https://www.rabbitmq.com/release-information
Node name: rabbit@rabbitmq-nordy-prod-server-2.rabbitmq-nordy-prod-nodes.db-prod
Erlang configuration: Erlang/OTP 26 [erts-14.2.5.4] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit:ns]
Crypto library: OpenSSL 3.1.7 3 Sep 2024
Erlang processes: 625 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /operator/enabled_plugins
Enabled plugins:

 * rabbitmq_mqtt
 * rabbitmq_prometheus
 * rabbitmq_federation
 * rabbitmq_peer_discovery_k8s
 * rabbitmq_top
 * accept
 * rabbitmq_peer_discovery_common
 * rabbitmq_shovel
 * amqp10_client
 * prometheus
 * rabbitmq_management
 * rabbitmq_management_agent
 * rabbitmq_web_dispatch
 * amqp_client
 * cowboy
 * cowlib
 * oauth2_client
 * jose

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@<kubernetes>
Raft data directory: /var/lib/rabbitmq/mnesia/rabbit@<kubernetes>/quorum/rabbit@<kubernetes>

Config files

 * /etc/rabbitmq/conf.d/10-defaults.conf
 * /etc/rabbitmq/conf.d/10-operatorDefaults.conf
 * /etc/rabbitmq/conf.d/11-default_user.conf
 * /etc/rabbitmq/conf.d/90-userDefinedConfiguration.conf

Log file(s)

 * /var/log/rabbitmq/rabbit@<kubernetes>.log
 * <stdout>

Alarms

(none)

Memory

Total memory used: 0.3658 gb
Calculation strategy: allocated
Memory high watermark setting: 0.9 gb, computed to: 0.8389 gb

quorum_ets: 0.1197 gb (27.25 %)
binary: 0.1085 gb (24.7 %)
reserved_unallocated: 0.0735 gb (16.74 %)
code: 0.0411 gb (9.35 %)
other_system: 0.0273 gb (6.21 %)
other_proc: 0.0251 gb (5.71 %)
allocated_unused: 0.0238 gb (5.42 %)
plugins: 0.0082 gb (1.86 %)
other_ets: 0.0059 gb (1.34 %)
atom: 0.0019 gb (0.43 %)
metrics: 0.0011 gb (0.26 %)
mgmt_db: 0.001 gb (0.22 %)
connection_other: 0.0009 gb (0.2 %)
quorum_queue_procs: 0.0006 gb (0.14 %)
connection_readers: 0.0003 gb (0.07 %)
msg_index: 0.0002 gb (0.05 %)
mnesia: 0.0001 gb (0.02 %)
metadata_store: 0.0001 gb (0.02 %)
connection_channels: 0.0001 gb (0.01 %)
metadata_store_ets: 0.0 gb (0.01 %)
connection_writers: 0.0 gb (0.0 %)
quorum_queue_dlx_procs: 0.0 gb (0.0 %)
stream_queue_procs: 0.0 gb (0.0 %)
stream_queue_replica_reader_procs: 0.0 gb (0.0 %)
queue_procs: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
stream_queue_coordinator_procs: 0.0 gb (0.0 %)

File Descriptors

Total: 25, limit: 1048479
Sockets: 11, limit: 943629

Free Disk Space

Low free disk space watermark: 2.0 gb
Free disk space: 20.7361 gb

Totals

Connection count: 11
Queue count: 4
Virtual host count: 1

Listeners

Interface: [::], port: 15672, protocol: http, purpose: HTTP API
Interface: [::], port: 15692, protocol: http/prometheus, purpose: Prometheus exporter API over HTTP
Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0
Interface: [::], port: 1883, protocol: mqtt, purpose: MQTT

Logs from node 1 (with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

rabbitmq 2024-10-24 21:15:57.848958+00:00 [info] <0.340212.0> accepting AMQP connection <0.340212.0> (3.39.145.193:1868 -> 10.200.101.159:5672)
rabbitmq 2024-10-24 21:15:57.901377+00:00 [info] <0.340212.0> connection <0.340212.0> (3.39.145.193:1868 -> 10.200.101.159:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 21:20:53.546397+00:00 [warning] <0.340212.0> closing AMQP connection <0.340212.0> (3.39.145.193:1868 -> 10.200.101.159:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 21:20:53.546397+00:00 [warning] <0.340212.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 22:07:17.653448+00:00 [info] <0.356717.0> accepting AMQP connection <0.356717.0> (3.39.145.193:36993 -> 10.200.101.159:5672)
rabbitmq 2024-10-24 22:07:17.712914+00:00 [info] <0.356717.0> connection <0.356717.0> (3.39.145.193:36993 -> 10.200.101.159:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 22:12:27.988829+00:00 [warning] <0.356717.0> closing AMQP connection <0.356717.0> (3.39.145.193:36993 -> 10.200.101.159:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 22:12:27.988829+00:00 [warning] <0.356717.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 22:25:25.129247+00:00 [info] <0.362544.0> accepting AMQP connection <0.362544.0> (211.193.38.6:33814 -> 10.200.101.159:5672)
rabbitmq 2024-10-24 22:25:25.143788+00:00 [info] <0.362544.0> connection <0.362544.0> (211.193.38.6:33814 -> 10.200.101.159:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-25 00:23:41.927085+00:00 [info] <0.401197.0> MQTT: discovered cluster peers that support client ID tracking: [{mqtt_node,
rabbitmq 2024-10-25 00:23:41.927085+00:00 [info] <0.401197.0>                                                                   'rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod'},
rabbitmq 2024-10-25 00:23:41.927085+00:00 [info] <0.401197.0>                                                                  {mqtt_node,
rabbitmq 2024-10-25 00:23:41.927085+00:00 [info] <0.401197.0>                                                                   'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod'}]
rabbitmq 2024-10-25 00:23:41.935968+00:00 [info] <0.401197.0> MQTT: starting Ra server with initial members: [{mqtt_node,
rabbitmq 2024-10-25 00:23:41.935968+00:00 [info] <0.401197.0>                                                  'rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod'},
rabbitmq 2024-10-25 00:23:41.935968+00:00 [info] <0.401197.0>                                                 {mqtt_node,
rabbitmq 2024-10-25 00:23:41.935968+00:00 [info] <0.401197.0>                                                  'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod'}]
rabbitmq 2024-10-25 00:23:41.954899+00:00 [info] <0.401203.0> Starting MQTT retained message store rabbit_mqtt_retained_msg_store_dets for vhost '/'
rabbitmq 2024-10-25 00:23:41.955515+00:00 [info] <0.401203.0> Recovered MQTT retained message store rabbit_mqtt_retained_msg_store_dets for vhost '/'
rabbitmq 2024-10-25 00:23:41.956397+00:00 [info] <0.401223.0> started MQTT TCP listener on :1883
rabbitmq 2024-10-25 00:23:41.957284+00:00 [info] <0.401180.0> Plugins changed; enabled [rabbitmq_mqtt]
rabbitmq 2024-10-25 00:24:43.018589+00:00 [notice] <0.401199.0> mqtt_node: candidate -> leader in term: 1 machine version: 1
rabbitmq 2024-10-25 00:25:13.265815+00:00 [info] <0.401779.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:25:13.266088+00:00 [info] <0.401779.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:30:05.203603+00:00 [info] <0.403399.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:30:05.203768+00:00 [info] <0.403399.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:30:20.276472+00:00 [info] <0.403527.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:30:20.276668+00:00 [info] <0.403527.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:30:22.980522+00:00 [info] <0.403565.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:30:22.980726+00:00 [info] <0.403565.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:30:29.175960+00:00 [info] <0.403634.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:30:29.176185+00:00 [info] <0.403634.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:31:20.291767+00:00 [info] <0.403953.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:31:20.291940+00:00 [info] <0.403953.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:31:21.367467+00:00 [info] <0.403981.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:31:21.367638+00:00 [info] <0.403981.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:31:22.315484+00:00 [info] <0.404012.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:31:22.317875+00:00 [info] <0.404012.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:31:24.094430+00:00 [info] <0.404054.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:31:24.094613+00:00 [info] <0.404054.0> Successfully synced tables from a peer
Stream closed EOF for db-prod/<kubernetes>-server-0 (setup-container)
rabbitmq 2024-10-25 00:31:45.132128+00:00 [info] <0.404214.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:31:45.132322+00:00 [info] <0.404214.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:33:36.297085+00:00 [info] <0.404936.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:33:36.297415+00:00 [info] <0.404936.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:33:37.819790+00:00 [info] <0.404973.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:33:37.820054+00:00 [info] <0.404973.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:33:41.517618+00:00 [info] <0.405019.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:33:41.517784+00:00 [info] <0.405019.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:33:42.520000+00:00 [info] <0.405051.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:33:42.520165+00:00 [info] <0.405051.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:34:32.329811+00:00 [notice] <0.403.0> Feature flags: attempt to enable `delete_ra_cluster_mqtt_node`...
rabbitmq 2024-10-25 00:34:32.489366+00:00 [info] <0.405415.0> Trying to acquire lock {mqtt_node,
rabbitmq 2024-10-25 00:34:32.489366+00:00 [info] <0.405415.0>                            {mqtt_node,
rabbitmq 2024-10-25 00:34:32.489366+00:00 [info] <0.405415.0>                                'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'}} on nodes ['rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod',
rabbitmq 2024-10-25 00:34:32.489366+00:00 [info] <0.405415.0>                                                                                                                    'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod',
rabbitmq 2024-10-25 00:34:32.489366+00:00 [info] <0.405415.0>                                                                                                                    'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'] ...
rabbitmq 2024-10-25 00:34:32.488984+00:00 [info] <15478.381614.0> Trying to acquire lock {mqtt_node,
rabbitmq 2024-10-25 00:34:32.488984+00:00 [info] <15478.381614.0>                            {mqtt_node,
rabbitmq 2024-10-25 00:34:32.488984+00:00 [info] <15478.381614.0>                                'rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod'}} on nodes ['rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod',
rabbitmq 2024-10-25 00:34:32.488984+00:00 [info] <15478.381614.0>                                                                                                                    'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod',
rabbitmq 2024-10-25 00:34:32.488984+00:00 [info] <15478.381614.0>                                                                                                                    'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'] ...
rabbitmq 2024-10-25 00:34:32.489710+00:00 [info] <15479.385892.0> Trying to acquire lock {mqtt_node,
rabbitmq 2024-10-25 00:34:32.489710+00:00 [info] <15479.385892.0>                            {mqtt_node,
rabbitmq 2024-10-25 00:34:32.489710+00:00 [info] <15479.385892.0>                                'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod'}} on nodes ['rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod',
rabbitmq 2024-10-25 00:34:32.489710+00:00 [info] <15479.385892.0>                                                                                                                    'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod',
rabbitmq 2024-10-25 00:34:32.489710+00:00 [info] <15479.385892.0>                                                                                                                    'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'] ...
rabbitmq 2024-10-25 00:34:32.517481+00:00 [info] <15478.381614.0> Acquired lock {mqtt_node,{mqtt_node,'rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod'}}
rabbitmq 2024-10-25 00:34:32.517646+00:00 [info] <15478.381614.0> Deleting Ra cluster mqtt_node ...
rabbitmq 2024-10-25 00:34:32.523806+00:00 [notice] <0.401199.0> mqtt_node: leader -> terminating_leader in term: 1 machine version: 1
rabbitmq 2024-10-25 00:34:32.524675+00:00 [info] <15478.381614.0> Successfully deleted Ra cluster mqtt_node
rabbitmq 2024-10-25 00:34:32.528182+00:00 [notice] <0.401199.0> mqtt_node: terminating with reason 'delete'
rabbitmq 2024-10-25 00:34:32.527870+00:00 [info] <15478.381614.0> Released lock {mqtt_node,{mqtt_node,'rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod'}}
rabbitmq 2024-10-25 00:34:32.605135+00:00 [info] <0.405415.0> Acquired lock {mqtt_node,{mqtt_node,'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'}}
rabbitmq 2024-10-25 00:34:32.605423+00:00 [info] <0.405415.0> Local Ra process mqtt_node does not exist
rabbitmq 2024-10-25 00:34:32.607413+00:00 [info] <0.405415.0> Released lock {mqtt_node,{mqtt_node,'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'}}
rabbitmq 2024-10-25 00:34:32.693208+00:00 [info] <15479.385892.0> Acquired lock {mqtt_node,{mqtt_node,'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod'}}
rabbitmq 2024-10-25 00:34:32.693365+00:00 [info] <15479.385892.0> Local Ra process mqtt_node does not exist
rabbitmq 2024-10-25 00:34:32.695071+00:00 [info] <15479.385892.0> Released lock {mqtt_node,{mqtt_node,'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod'}}
rabbitmq 2024-10-25 00:34:32.839020+00:00 [notice] <0.403.0> Feature flags: `delete_ra_cluster_mqtt_node` enabled
rabbitmq 2024-10-25 00:34:32.970980+00:00 [info] <0.405466.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:34:32.971325+00:00 [info] <0.405466.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:34:34.455185+00:00 [info] <0.405526.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:34:34.455380+00:00 [info] <0.405526.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:34:35.968851+00:00 [info] <0.405582.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:34:35.969008+00:00 [info] <0.405582.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:35:00.051433+00:00 [info] <0.405781.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:35:00.051610+00:00 [info] <0.405781.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:35:00.743445+00:00 [info] <0.405786.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:35:00.743650+00:00 [info] <0.405786.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:35:00.818273+00:00 [info] <0.405805.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:35:00.818448+00:00 [info] <0.405805.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:35:04.528046+00:00 [info] <0.405859.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:35:04.529014+00:00 [info] <0.405859.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:35:05.841378+00:00 [info] <0.405887.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:35:05.841648+00:00 [info] <0.405887.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:35:06.870493+00:00 [info] <0.405919.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:35:06.870673+00:00 [info] <0.405919.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:35:09.201341+00:00 [info] <0.405958.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:35:09.201547+00:00 [info] <0.405958.0> Successfully synced tables from a peer

Logs from node 2 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

rabbitmq 2024-10-24 09:29:03.667282+00:00 [info] <0.102261.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 09:49:40.252798+00:00 [info] <0.109096.0> accepting AMQP connection <0.109096.0> (3.39.145.193:36829 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 09:49:40.306738+00:00 [info] <0.109096.0> connection <0.109096.0> (3.39.145.193:36829 -> 10.200.71.72:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 09:49:40.675150+00:00 [info] <0.109114.0> accepting AMQP connection <0.109114.0> (3.39.145.193:24124 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 09:49:40.730634+00:00 [info] <0.109114.0> connection <0.109114.0> (3.39.145.193:24124 -> 10.200.71.72:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 09:52:06.714355+00:00 [info] <0.109932.0> accepting AMQP connection <0.109932.0> (3.39.145.193:6125 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 09:52:06.774589+00:00 [info] <0.109932.0> connection <0.109932.0> (3.39.145.193:6125 -> 10.200.71.72:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 09:54:35.358749+00:00 [warning] <0.109932.0> closing AMQP connection <0.109932.0> (3.39.145.193:6125 -> 10.200.71.72:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 09:54:35.358749+00:00 [warning] <0.109932.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 09:54:35.359185+00:00 [warning] <0.109114.0> closing AMQP connection <0.109114.0> (3.39.145.193:24124 -> 10.200.71.72:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 09:54:35.359185+00:00 [warning] <0.109114.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 09:54:50.363495+00:00 [warning] <0.109096.0> closing AMQP connection <0.109096.0> (3.39.145.193:36829 -> 10.200.71.72:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 09:54:50.363495+00:00 [warning] <0.109096.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 10:17:34.871013+00:00 [info] <0.118475.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 10:17:34.871190+00:00 [info] <0.118475.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 10:45:05.474895+00:00 [info] <0.128415.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 10:45:05.475073+00:00 [info] <0.128415.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 10:51:54.233259+00:00 [info] <0.130942.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 10:51:54.233558+00:00 [info] <0.130942.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 11:02:16.862694+00:00 [info] <0.134580.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 11:02:16.862878+00:00 [info] <0.134580.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 11:07:07.089683+00:00 [info] <0.136358.0> accepting AMQP connection <0.136358.0> (172.168.159.43:45964 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 11:07:07.089902+00:00 [error] <0.136358.0> closing AMQP connection <0.136358.0> (172.168.159.43:45964 -> 10.200.71.72:5672):
rabbitmq 2024-10-24 11:07:07.089902+00:00 [error] <0.136358.0> {bad_header,<<"MGLNDD_1">>}
rabbitmq 2024-10-24 11:09:02.919777+00:00 [info] <0.137102.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 11:09:02.919998+00:00 [info] <0.137102.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 11:26:11.588658+00:00 [info] <0.143175.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 11:26:11.588851+00:00 [info] <0.143175.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 11:36:38.245870+00:00 [info] <0.147035.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 11:36:38.246090+00:00 [info] <0.147035.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 11:57:00.829042+00:00 [warning] <0.98908.0> closing AMQP connection <0.98908.0> (52.78.48.219:41198 -> 10.200.71.72:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 11:57:00.829042+00:00 [warning] <0.98908.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 11:57:00.829310+00:00 [warning] <0.98195.0> closing AMQP connection <0.98195.0> (52.78.48.219:3639 -> 10.200.71.72:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 11:57:00.829310+00:00 [warning] <0.98195.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 11:57:06.942040+00:00 [info] <0.154081.0> accepting AMQP connection <0.154081.0> (52.78.48.219:28058 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 11:57:07.007406+00:00 [info] <0.154081.0> connection <0.154081.0> (52.78.48.219:28058 -> 10.200.71.72:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 12:00:30.811145+00:00 [info] <0.155308.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 12:00:30.811408+00:00 [info] <0.155308.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 12:17:40.644289+00:00 [info] <0.161118.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 12:17:40.645931+00:00 [info] <0.161118.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 18:18:40.800408+00:00 [info] <0.268732.0> accepting AMQP connection <0.268732.0> (106.75.29.243:57498 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 18:18:40.917652+00:00 [warning] <0.268732.0> closing AMQP connection <0.268732.0> (106.75.29.243:57498 -> 10.200.71.72:5672):
rabbitmq 2024-10-24 18:18:40.917652+00:00 [warning] <0.268732.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 18:26:38.543437+00:00 [info] <0.271095.0> accepting AMQP connection <0.271095.0> (205.210.31.23:58036 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 18:26:38.719805+00:00 [warning] <0.271095.0> closing AMQP connection <0.271095.0> (205.210.31.23:58036 -> 10.200.71.72:5672):
rabbitmq 2024-10-24 18:26:38.719805+00:00 [warning] <0.271095.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 18:39:47.363287+00:00 [info] <0.275010.0> accepting AMQP connection <0.275010.0> (3.39.145.193:32782 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 18:39:47.418805+00:00 [info] <0.275010.0> connection <0.275010.0> (3.39.145.193:32782 -> 10.200.71.72:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 19:39:01.886767+00:00 [info] <0.292635.0> accepting AMQP connection <0.292635.0> (205.210.31.223:59626 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 19:39:02.179900+00:00 [warning] <0.292635.0> closing AMQP connection <0.292635.0> (205.210.31.223:59626 -> 10.200.71.72:5672):
rabbitmq 2024-10-24 19:39:02.179900+00:00 [warning] <0.292635.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 22:07:17.368945+00:00 [info] <0.336699.0> accepting AMQP connection <0.336699.0> (3.39.145.193:7323 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 22:07:17.438831+00:00 [info] <0.336699.0> connection <0.336699.0> (3.39.145.193:7323 -> 10.200.71.72:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 22:07:17.662140+00:00 [info] <0.336712.0> accepting AMQP connection <0.336712.0> (3.39.145.193:33728 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 22:07:17.718828+00:00 [info] <0.336712.0> connection <0.336712.0> (3.39.145.193:33728 -> 10.200.71.72:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 22:12:27.989686+00:00 [warning] <0.336712.0> closing AMQP connection <0.336712.0> (3.39.145.193:33728 -> 10.200.71.72:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 22:12:27.989686+00:00 [warning] <0.336712.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 22:12:27.996015+00:00 [warning] <0.336699.0> closing AMQP connection <0.336699.0> (3.39.145.193:7323 -> 10.200.71.72:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 22:12:27.996015+00:00 [warning] <0.336699.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 22:42:42.822904+00:00 [info] <0.347267.0> accepting AMQP connection <0.347267.0> (211.193.38.6:48580 -> 10.200.71.72:5672)
rabbitmq 2024-10-24 22:42:42.833270+00:00 [info] <0.347267.0> connection <0.347267.0> (211.193.38.6:48580 -> 10.200.71.72:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-25 00:24:41.654766+00:00 [info] <0.378232.0> MQTT: discovered cluster peers that support client ID tracking: [{mqtt_node,
rabbitmq 2024-10-25 00:24:41.654766+00:00 [info] <0.378232.0>                                                                   'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod'},
rabbitmq 2024-10-25 00:24:41.654766+00:00 [info] <0.378232.0>                                                                  {mqtt_node,
rabbitmq 2024-10-25 00:24:41.654766+00:00 [info] <0.378232.0>                                                                   'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'}]
rabbitmq 2024-10-25 00:24:41.661608+00:00 [info] <0.378232.0> MQTT: starting Ra server with initial members: [{mqtt_node,
rabbitmq 2024-10-25 00:24:41.661608+00:00 [info] <0.378232.0>                                                  'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod'},
rabbitmq 2024-10-25 00:24:41.661608+00:00 [info] <0.378232.0>                                                 {mqtt_node,
rabbitmq 2024-10-25 00:24:41.661608+00:00 [info] <0.378232.0>                                                  'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'}]
rabbitmq 2024-10-25 00:24:43.010781+00:00 [info] <0.378234.0> mqtt_node: granting vote for {mqtt_node,'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'} with last indexterm {0,0} for term 1 previous term was 0
rabbitmq 2024-10-25 00:24:43.019281+00:00 [info] <0.378234.0> mqtt_node: detected a new leader {mqtt_node,'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'} in term 1
rabbitmq 2024-10-25 00:24:43.020539+00:00 [info] <0.378238.0> Starting MQTT retained message store rabbit_mqtt_retained_msg_store_dets for vhost '/'
rabbitmq 2024-10-25 00:24:43.021125+00:00 [info] <0.378238.0> Recovered MQTT retained message store rabbit_mqtt_retained_msg_store_dets for vhost '/'
rabbitmq 2024-10-25 00:24:43.022639+00:00 [info] <0.378259.0> started MQTT TCP listener on :1883
rabbitmq 2024-10-25 00:24:43.023587+00:00 [info] <0.378215.0> Plugins changed; enabled [rabbitmq_mqtt]
rabbitmq 2024-10-25 00:24:43.027672+00:00 [info] <0.378234.0> mqtt_node: detected a new leader {mqtt_node,'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'} in term 1
rabbitmq 2024-10-25 00:32:53.488484+00:00 [info] <0.380900.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:32:53.488770+00:00 [info] <0.380900.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:32:55.757710+00:00 [info] <0.380945.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:32:55.757877+00:00 [info] <0.380945.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:34:19.385254+00:00 [info] <0.381462.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:34:19.385440+00:00 [info] <0.381462.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:34:23.223027+00:00 [info] <0.381504.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:34:23.223208+00:00 [info] <0.381504.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:34:25.188177+00:00 [info] <0.381566.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:34:25.188367+00:00 [info] <0.381566.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:34:32.488984+00:00 [info] <0.381614.0> Trying to acquire lock {mqtt_node,
rabbitmq 2024-10-25 00:34:32.488984+00:00 [info] <0.381614.0>                            {mqtt_node,
rabbitmq 2024-10-25 00:34:32.488984+00:00 [info] <0.381614.0>                                'rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod'}} on nodes ['rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod',
rabbitmq 2024-10-25 00:34:32.488984+00:00 [info] <0.381614.0>                                                                                                                    'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod',
rabbitmq 2024-10-25 00:34:32.488984+00:00 [info] <0.381614.0>                                                                                                                    'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'] ...
rabbitmq 2024-10-25 00:34:32.517481+00:00 [info] <0.381614.0> Acquired lock {mqtt_node,{mqtt_node,'rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod'}}
rabbitmq 2024-10-25 00:34:32.517646+00:00 [info] <0.381614.0> Deleting Ra cluster mqtt_node ...
rabbitmq 2024-10-25 00:34:32.524675+00:00 [info] <0.381614.0> Successfully deleted Ra cluster mqtt_node
rabbitmq 2024-10-25 00:34:32.527870+00:00 [info] <0.381614.0> Released lock {mqtt_node,{mqtt_node,'rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod'}}
rabbitmq 2024-10-25 00:34:32.528820+00:00 [notice] <0.378234.0> mqtt_node: terminating with reason 'delete'
rabbitmq 2024-10-25 00:34:46.725229+00:00 [info] <0.381795.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:34:46.725418+00:00 [info] <0.381795.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:34:47.892496+00:00 [info] <0.381824.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:34:47.892725+00:00 [info] <0.381824.0> Successfully synced tables from a peer
Stream closed EOF for db-prod/<kubernetes>-server-1 (setup-container)

Logs from node 3 (if applicable, with sensitive values edited out)

See https://www.rabbitmq.com/docs/logging to learn how to collect logs

rabbitmq 2024-10-24 11:53:49.764048+00:00 [info] <0.156155.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 11:57:00.829366+00:00 [warning] <0.102013.0> closing AMQP connection <0.102013.0> (52.78.48.219:1206 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 11:57:00.829366+00:00 [warning] <0.102013.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 11:57:26.745802+00:00 [info] <0.157488.0> accepting AMQP connection <0.157488.0> (52.78.48.219:4793 -> 10.200.84.115:5672)
rabbitmq 2024-10-24 11:57:26.796021+00:00 [info] <0.157488.0> connection <0.157488.0> (52.78.48.219:4793 -> 10.200.84.115:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 12:11:02.942544+00:00 [info] <0.162421.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 12:11:02.942734+00:00 [info] <0.162421.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 18:39:42.760761+00:00 [warning] <0.1941.0> closing AMQP connection <0.1941.0> (3.39.145.193:46794 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 18:39:42.760761+00:00 [warning] <0.1941.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 18:39:42.761724+00:00 [warning] <0.1871.0> closing AMQP connection <0.1871.0> (3.39.145.193:31774 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 18:39:42.761724+00:00 [warning] <0.1871.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 18:39:42.761503+00:00 [warning] <0.1829.0> closing AMQP connection <0.1829.0> (3.39.145.193:2465 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 18:39:42.761503+00:00 [warning] <0.1829.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 18:39:47.364097+00:00 [info] <0.278606.0> accepting AMQP connection <0.278606.0> (3.39.145.193:63224 -> 10.200.84.115:5672)
rabbitmq 2024-10-24 18:39:47.423157+00:00 [info] <0.278606.0> connection <0.278606.0> (3.39.145.193:63224 -> 10.200.84.115:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 18:40:38.840533+00:00 [info] <0.278879.0> accepting AMQP connection <0.278879.0> (3.39.145.193:61907 -> 10.200.84.115:5672)
rabbitmq 2024-10-24 18:40:38.892063+00:00 [info] <0.278879.0> connection <0.278879.0> (3.39.145.193:61907 -> 10.200.84.115:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 18:51:54.106560+00:00 [warning] <0.1842.0> closing AMQP connection <0.1842.0> (13.124.10.113:26681 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 18:51:54.106560+00:00 [warning] <0.1842.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 18:51:54.107927+00:00 [warning] <0.1865.0> closing AMQP connection <0.1865.0> (13.124.10.113:41299 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 18:51:54.107927+00:00 [warning] <0.1865.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 18:51:54.107941+00:00 [warning] <0.1838.0> closing AMQP connection <0.1838.0> (13.124.10.113:63895 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 18:51:54.107941+00:00 [warning] <0.1838.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 18:51:58.918050+00:00 [info] <0.282274.0> accepting AMQP connection <0.282274.0> (13.124.10.113:43350 -> 10.200.84.115:5672)
rabbitmq 2024-10-24 18:51:59.002121+00:00 [info] <0.282274.0> connection <0.282274.0> (13.124.10.113:43350 -> 10.200.84.115:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 21:15:57.850149+00:00 [info] <0.325075.0> accepting AMQP connection <0.325075.0> (3.39.145.193:7844 -> 10.200.84.115:5672)
rabbitmq 2024-10-24 21:15:57.907274+00:00 [info] <0.325075.0> connection <0.325075.0> (3.39.145.193:7844 -> 10.200.84.115:5672): user 'admin' authenticated and granted access to vhost '/'
Stream closed EOF for db-prod/<kubernetes>-server-2 (setup-container)
rabbitmq 2024-10-24 21:18:01.345804+00:00 [info] <0.325698.0> accepting AMQP connection <0.325698.0> (3.39.145.193:15788 -> 10.200.84.115:5672)
rabbitmq 2024-10-24 21:18:01.399158+00:00 [info] <0.325698.0> connection <0.325698.0> (3.39.145.193:15788 -> 10.200.84.115:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 21:20:53.546975+00:00 [warning] <0.325698.0> closing AMQP connection <0.325698.0> (3.39.145.193:15788 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 21:20:53.546975+00:00 [warning] <0.325698.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 21:20:53.546925+00:00 [warning] <0.325075.0> closing AMQP connection <0.325075.0> (3.39.145.193:7844 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 21:20:53.546925+00:00 [warning] <0.325075.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 22:07:17.368539+00:00 [info] <0.340363.0> accepting AMQP connection <0.340363.0> (3.39.145.193:39783 -> 10.200.84.115:5672)
rabbitmq 2024-10-24 22:07:17.439063+00:00 [info] <0.340363.0> connection <0.340363.0> (3.39.145.193:39783 -> 10.200.84.115:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 22:08:49.197174+00:00 [info] <0.340834.0> accepting AMQP connection <0.340834.0> (3.39.145.193:33262 -> 10.200.84.115:5672)
rabbitmq 2024-10-24 22:08:49.263580+00:00 [info] <0.340834.0> connection <0.340834.0> (3.39.145.193:33262 -> 10.200.84.115:5672): user 'admin' authenticated and granted access to vhost '/'
rabbitmq 2024-10-24 22:12:27.996016+00:00 [warning] <0.340834.0> closing AMQP connection <0.340834.0> (3.39.145.193:33262 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 22:12:27.996016+00:00 [warning] <0.340834.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 22:12:27.996217+00:00 [warning] <0.340363.0> closing AMQP connection <0.340363.0> (3.39.145.193:39783 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 22:12:27.996217+00:00 [warning] <0.340363.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 23:28:10.577254+00:00 [warning] <0.15295.0> closing AMQP connection <0.15295.0> (211.193.38.6:36216 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 23:28:10.577254+00:00 [warning] <0.15295.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 23:32:32.721576+00:00 [warning] <0.14895.0> closing AMQP connection <0.14895.0> (211.193.38.6:45096 -> 10.200.84.115:5672, vhost: '/', user: 'admin'):
rabbitmq 2024-10-24 23:32:32.721576+00:00 [warning] <0.14895.0> client unexpectedly closed TCP connection
rabbitmq 2024-10-24 23:59:42.907724+00:00 [info] <0.373834.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 23:59:42.907903+00:00 [info] <0.373834.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 23:59:52.155490+00:00 [info] <0.373902.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 23:59:52.155701+00:00 [info] <0.373902.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 23:59:55.064363+00:00 [info] <0.373934.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 23:59:55.064536+00:00 [info] <0.373934.0> Successfully synced tables from a peer
rabbitmq 2024-10-24 23:59:56.774859+00:00 [info] <0.373956.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-24 23:59:56.775124+00:00 [info] <0.373956.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:00:03.349208+00:00 [info] <0.374009.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:00:03.349392+00:00 [info] <0.374009.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:00:16.233983+00:00 [info] <0.374094.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:00:16.234356+00:00 [info] <0.374094.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:00:17.758709+00:00 [info] <0.374109.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:00:17.758900+00:00 [info] <0.374109.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:00:18.828741+00:00 [info] <0.374131.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:00:18.828923+00:00 [info] <0.374131.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:00:19.857237+00:00 [info] <0.374146.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:00:19.857412+00:00 [info] <0.374146.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:00:20.517870+00:00 [info] <0.374158.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:00:20.518045+00:00 [info] <0.374158.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:00:22.016810+00:00 [info] <0.374174.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:00:22.017001+00:00 [info] <0.374174.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:08:56.545760+00:00 [info] <0.377038.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:08:56.546114+00:00 [info] <0.377038.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:20:44.672895+00:00 [info] <0.381216.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:20:44.673107+00:00 [info] <0.381216.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:20:48.795217+00:00 [info] <0.381250.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:20:48.795705+00:00 [info] <0.381250.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:20:50.690530+00:00 [info] <0.381266.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:20:50.690704+00:00 [info] <0.381266.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:24:53.444471+00:00 [info] <0.382662.0> MQTT: discovered cluster peers that support client ID tracking: [{mqtt_node,
rabbitmq 2024-10-25 00:24:53.444471+00:00 [info] <0.382662.0>                                                                   'rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod'},
rabbitmq 2024-10-25 00:24:53.444471+00:00 [info] <0.382662.0>                                                                  {mqtt_node,
rabbitmq 2024-10-25 00:24:53.444471+00:00 [info] <0.382662.0>                                                                   'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'}]
rabbitmq 2024-10-25 00:24:53.451296+00:00 [info] <0.382662.0> MQTT: starting Ra server with initial members: [{mqtt_node,
rabbitmq 2024-10-25 00:24:53.451296+00:00 [info] <0.382662.0>                                                  'rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod'},
rabbitmq 2024-10-25 00:24:53.451296+00:00 [info] <0.382662.0>                                                 {mqtt_node,
rabbitmq 2024-10-25 00:24:53.451296+00:00 [info] <0.382662.0>                                                  'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'}]
rabbitmq 2024-10-25 00:24:53.468853+00:00 [info] <0.382668.0> Starting MQTT retained message store rabbit_mqtt_retained_msg_store_dets for vhost '/'
rabbitmq 2024-10-25 00:24:53.469805+00:00 [info] <0.382668.0> Recovered MQTT retained message store rabbit_mqtt_retained_msg_store_dets for vhost '/'
rabbitmq 2024-10-25 00:24:53.479220+00:00 [info] <0.382690.0> started MQTT TCP listener on :1883
rabbitmq 2024-10-25 00:24:53.480257+00:00 [info] <0.382645.0> Plugins changed; enabled [rabbitmq_mqtt]
rabbitmq 2024-10-25 00:24:53.484574+00:00 [info] <0.382664.0> mqtt_node: follower did not have entry at 1 in 1. Requesting {mqtt_node,'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'} from 1
rabbitmq 2024-10-25 00:24:53.484755+00:00 [info] <0.382664.0> mqtt_node: detected a new leader {mqtt_node,'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'} in term 1
rabbitmq 2024-10-25 00:25:07.152667+00:00 [info] <0.382785.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
rabbitmq 2024-10-25 00:25:07.153310+00:00 [info] <0.382785.0> Successfully synced tables from a peer
rabbitmq 2024-10-25 00:34:32.489710+00:00 [info] <0.385892.0> Trying to acquire lock {mqtt_node,
rabbitmq 2024-10-25 00:34:32.489710+00:00 [info] <0.385892.0>                            {mqtt_node,
rabbitmq 2024-10-25 00:34:32.489710+00:00 [info] <0.385892.0>                                'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod'}} on nodes ['rabbit@<kubernetes>-server-1.<kubernetes>-nodes.db-prod',
rabbitmq 2024-10-25 00:34:32.489710+00:00 [info] <0.385892.0>                                                                                                                    'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod',
rabbitmq 2024-10-25 00:34:32.489710+00:00 [info] <0.385892.0>                                                                                                                    'rabbit@<kubernetes>-server-0.<kubernetes>-nodes.db-prod'] ...
rabbitmq 2024-10-25 00:34:32.528796+00:00 [notice] <0.382664.0> mqtt_node: terminating with reason 'delete'
rabbitmq 2024-10-25 00:34:32.693208+00:00 [info] <0.385892.0> Acquired lock {mqtt_node,{mqtt_node,'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod'}}
rabbitmq 2024-10-25 00:34:32.693365+00:00 [info] <0.385892.0> Local Ra process mqtt_node does not exist
rabbitmq 2024-10-25 00:34:32.695071+00:00 [info] <0.385892.0> Released lock {mqtt_node,{mqtt_node,'rabbit@<kubernetes>-server-2.<kubernetes>-nodes.db-prod'}}

rabbitmq.conf

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find rabbitmq.conf file location

# PASTE rabbitmq.conf HERE, BETWEEN BACKTICKS

Steps to deploy RabbitMQ cluster

I Use Kubernetes Operator to deploy rabbitmq-cluster. (cluster-operator:2.11.0)
We first built our cluster on rabbitmq:3.10.2 and have been going through 3.11 and 3.12 to upgrade to 3.13.7.

Steps to reproduce the behavior in question

we load this json on startup
(permission and users are removed)

  "policies": [
    {
      "vhost": "/",
      "name": "dlx-ttl-1d-limit-3",
      "pattern": ".*",
      "definition": {
        "dead-letter-exchange": "dlx",
        "message-ttl": 86400000,
        "delivery-limit": 3
      },
      "priority": 0,
      "apply-to": "queues"
    }
  ],
  "exchanges": [
    {
      "name": "dlx",
      "vhost": "/",
      "type": "topic",
      "durable": true,
      "internal": false,
      "arguments": {}
    }
  ],
  "queues": [
    {
      "name": "connect-request",
      "vhost": "/",
      "durable": true,
      "arguments": {
        "x-queue-type": "quorum"
      }
    },
    {
      "name": "connect-response",
      "vhost": "/",
      "durable": true,
      "arguments": {
        "x-queue-type": "quorum"
      }
    },
    {
      "name": "execute-response",
      "vhost": "/",
      "durable": true,
      "arguments": {
        "x-queue-type": "quorum"
      }
    },
    {
      "name": "dlx",
      "vhost": "/",
      "durable": true,
      "arguments": {
        "x-queue-type": "quorum"
      }
    }
  ],

advanced.config

See https://www.rabbitmq.com/docs/configure#config-location to learn how to find advanced.config file location

      load_definitions = /tmp/definitions.json
      # Connection 관리
      cluster_partition_handling = pause_minority
      heartbeat = 120
      tcp_listen_options.exit_on_close = true
      tcp_listen_options.nodelay = true
      tcp_listen_options.keepalive = true
      tcp_listen_options.send_timeout = 15000 

      # Connection cleanup  
      channel_max = 2000
      connection_max = 15000

      # Socket 설정
      consumer_timeout = 180000
      handshake_timeout = 10000

      # 메모리 관리
      vm_memory_high_watermark.absolute = 900MB 
      vm_memory_calculation_strategy = allocated 
      vm_memory_high_watermark_paging_ratio = 0.8
      disk_free_limit.relative = 1.0
      collect_statistics_interval = 60000
      log.file.level = warning
      cluster_partition_handling = autoheal

Application code

# PASTE CODE HERE, BETWEEN BACKTICKS

Kubernetes deployment file

# yaml-language-server: $schema=https://crd.us-east-1.airsmed.io/crds/rabbitmqcluster
apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq-nordy-prod
spec:
  replicas: 3
  image: rabbitmq:3.13.7-management
  persistence:
    storageClassName: ebs-sc
    storage: 20Gi
  resources:
    requests:
      cpu: 100m
      memory: 1000Mi
    limits:
      # cpu: 1
      memory: 1000Mi
  rabbitmq:
    additionalConfig: |
      load_definitions = /tmp/definitions.json
      # Connection 관리
      cluster_partition_handling = pause_minority
      heartbeat = 120
      tcp_listen_options.exit_on_close = true
      tcp_listen_options.nodelay = true
      tcp_listen_options.keepalive = true
      tcp_listen_options.send_timeout = 15000 

      # Connection cleanup  
      channel_max = 2000
      connection_max = 15000

      # Socket 설정
      consumer_timeout = 180000
      handshake_timeout = 10000

      # 메모리 관리
      vm_memory_high_watermark.absolute = 900MB 
      vm_memory_calculation_strategy = allocated 
      vm_memory_high_watermark_paging_ratio = 0.8
      disk_free_limit.relative = 1.0
      collect_statistics_interval = 60000
      log.file.level = warning
      cluster_partition_handling = autoheal

    additionalPlugins:
      - rabbitmq_top
      - rabbitmq_shovel
  override:
    statefulSet:
      spec:
        template:
          spec:
            containers:
              - name: rabbitmq
                volumeMounts:
                  - name: definitions
                    subPath: definitions.json
                    mountPath: /tmp/definitions.json
            volumes:
              - name: definitions
                projected:
                  sources:
                    - secret:
                        name: rabbitmq-definitions
                        items:
                          - key: definitions.json
                            path: definitions.json

  affinity:
    podAntiAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
        - podAffinityTerm:
            topologyKey: kubernetes.io/hostname
          weight: 100

What problem are you trying to solve?

After searching, the expected behavior of my settings is that when a message is received, it is moved to disk after a period of time, so there seems to be no reason why it should be kept in RAM. However, ra_log_ets keeps growing in use and does not get smaller. rabbitmq does nothing until it reaches a high watermark and stops forwarding messages.

What settings might affect this issue?