proposer in unhealty state until all pending batches are submitted

[srene]

Summary:
With config: very small max skew time and high submission time and block creation time
Got: timeLastBlockProduced - lastBlockSubmitted >= max skew time, and then producer pause

---

Dec 12 05:23:07 ip-10-0-2-51 roller[249344]: time="Dec 12 05:23:07.000" level=info msg="Submitted batch to SL.[start height 13536 end height 13572]" module=block_manager
Dec 12 05:23:07 ip-10-0-2-51 roller[249344]: time="Dec 12 05:23:07.001" level=debug msg="Submitted a batch to both sub-layers.[n bytes consumed from pending 41354 pending after 62586 skew time 1m1.599464184s]" module=block_manager
Dec 12 05:23:07 ip-10-0-2-51 roller[249344]: time="Dec 12 05:23:07.001" level=debug msg="Added bytes produced to bytes pending submission counter.[bytes added 1118 pending 103940]" module=block_manager
Dec 12 05:23:07 ip-10-0-2-51 roller[249344]: time="Dec 12 05:23:07.128" level=debug msg="Added bytes produced to bytes pending submission counter.[bytes added 37154 pending 141094]" module=block_manager
Dec 12 05:23:11 ip-10-0-2-51 roller[249344]: time="Dec 12 05:23:11.185" level=debug msg="Submitted blob to DA successfully.[]" module=celestia
Dec 12 05:23:11 ip-10-0-2-51 roller[249344]: time="Dec 12 05:23:11.238" level=info msg="Submitted batch to DA.[start height 13573 end height 13628]" module=block_manager
Dec 12 05:23:12 ip-10-0-2-51 roller[249344]: time="Dec 12 05:23:12.192" level=error msg="Pausing block production until new batch is submitted.[Batch skew time 3m35.199839706s Max batch skew time 2m0s Last block in settlement time 2024-12-12 05:19:36.971802242 +0000 UTC]" module=block_manager
Dec 12 05:23:12 ip-10-0-2-51 roller[249344]: time="Dec 12 05:23:12.192" level=error msg="node is unhealthy[error Block production paused. Time between last block produced and last block submitted higher than max skew time: 2m0s last block in settlement time: 2024-12-12 05:19:36.971802242 +0000 UTC resource exhausted]"
Dec 12 05:25:14 ip-10-0-2-51 roller[249344]: time="Dec 12 05:25:14.410" level=info msg="Submitted batch to SL.[start height 13573 end height 13628]" module=block_manager
Dec 12 05:25:14 ip-10-0-2-51 roller[249344]: time="Dec 12 05:25:14.410" level=debug msg="Submitted a batch to both sub-layers.[n bytes consumed from pending 62586 pending after 37905 skew time 2m33.600375522s]" module=block_manager
Dec 12 05:25:18 ip-10-0-2-51 roller[249344]: time="Dec 12 05:25:18.221" level=debug msg="Submitted blob to DA successfully.[]" module=celestia
Dec 12 05:25:18 ip-10-0-2-51 roller[249344]: time="Dec 12 05:25:18.249" level=info msg="Submitted batch to DA.[start height 13629 end height 13630]" module=block_manager
Dec 12 05:27:18 ip-10-0-2-51 roller[249344]: time="Dec 12 05:27:18.308" level=info msg="Submitted batch to SL.[start height 13629 end height 13630]" module=block_manager
Dec 12 05:27:18 ip-10-0-2-51 roller[249344]: time="Dec 12 05:27:18.309" level=debug msg="Submitted a batch to both sub-layers.[n bytes consumed from pending 37905 pending after 0 skew time 0s]" module=block_manager
Dec 12 05:27:18 ip-10-0-2-51 roller[249344]: time="Dec 12 05:27:18.309" level=debug msg="Added bytes produced to bytes pending submission counter.[bytes added 751 pending 751]" module=block_manager
Dec 12 05:27:18 ip-10-0-2-51 roller[249344]: time="Dec 12 05:27:18.337" level=debug msg="Added bytes produced to bytes pending submission counter.[bytes added 3301 pending 4052]" module=block_manager
Dec 12 05:27:21 ip-10-0-2-51 roller[249344]: time="Dec 12 05:27:21.906" level=debug msg="Submitted blob to DA successfully.[]" module=celestia
Dec 12 05:27:21 ip-10-0-2-51 roller[249344]: time="Dec 12 05:27:21.924" level=info msg="Submitted batch to DA.[start height 13631 end height 13631]" module=block_manager
Dec 12 05:27:23 ip-10-0-2-51 roller[249344]: time="Dec 12 05:27:23.394" level=error msg="Pausing block production until new batch is submitted.[Batch skew time 4m11.199821582s Max batch skew time 2m0s Last block in settlement time 2024-12-12 05:23:12.171641948 +0000 UTC]" module=block_manager
Dec 12 05:27:23 ip-10-0-2-51 roller[249344]: time="Dec 12 05:27:23.394" level=error msg="node is unhealthy[error Block production paused. Time between last block produced and last block submitted higher than max skew time: 2m0s last block in settlement time: 2024-12-12 05:23:12.171641948 +0000 UTC resource exhausted]"
Dec 12 05:29:21 ip-10-0-2-51 roller[249344]: time="Dec 12 05:29:21.975" level=info msg="Submitted batch to SL.[start height 13631 end height 13631]" module=block_manager
Dec 12 05:29:21 ip-10-0-2-51 roller[249344]: time="Dec 12 05:29:21.975" level=debug msg="Submitted a batch to both sub-layers.[n bytes consumed from pending 3301 pending after 753 skew time 5.062249168s]" module=block_manager
Dec 12 05:29:21 ip-10-0-2-51 roller[249344]: time="Dec 12 05:29:21.975" level=debug msg="Added bytes produced to bytes pending submission counter.[bytes added 753 pending 1506]" module=block_manager
Dec 12 05:29:21 ip-10-0-2-51 roller[249344]: time="Dec 12 05:29:21.994" level=debug msg="Added bytes produced to bytes pending submission counter.[bytes added 753 pending 2259]" module=block_manager
Dec 12 05:29:25 ip-10-0-2-51 roller[249344]: time="Dec 12 05:29:25.509" level=debug msg="Submitted blob to DA successfully.[]" module=celestia
Dec 12 05:29:25 ip-10-0-2-51 roller[249344]: time="Dec 12 05:29:25.526" level=info msg="Submitted batch to DA.[start height 13632 end height 13633]" module=block_manager
Dec 12 05:30:22 ip-10-0-2-51 roller[249344]: time="Dec 12 05:30:22.194" level=error msg="Pausing block production until new batch is submitted.[Batch skew time 3m3.862490242s Max batch skew time 2m0s Last block in settlement time 2024-12-12 05:27:18.309214362 +0000 UTC]" module=block_manager
Dec 12 05:30:22 ip-10-0-2-51 roller[249344]: time="Dec 12 05:30:22.194" level=error msg="node is unhealthy[error Block production paused. Time between last block produced and last block submitted higher than max skew time: 2m0s last block in settlement time: 2024-12-12 05:27:18.309214362 +0000 UTC resource exhausted]"
Dec 12 05:31:25 ip-10-0-2-51 roller[249344]: time="Dec 12 05:31:25.595" level=info msg="Submitted batch to SL.[start height 13632 end height 13633]" module=block_manager
Dec 12 05:31:25 ip-10-0-2-51 roller[249344]: time="Dec 12 05:31:25.595" level=debug msg="Submitted a batch to both sub-layers.[n bytes consumed from pending 1506 pending after 752 skew time 1m0.195864202s]" module=block_manager
Dec 12 05:31:25 ip-10-0-2-51 roller[249344]: time="Dec 12 05:31:25.595" level=debug msg="Added bytes produced to bytes pending submission counter.[bytes added 752 pending 3011]" module=block_manager
Dec 12 05:31:25 ip-10-0-2-51 roller[249344]: time="Dec 12 05:31:25.617" level=debug msg="Added bytes produced to bytes pending submission counter.[bytes added 752 pending 3763]" module=block_manager
Dec 12 05:31:32 ip-10-0-2-51 roller[249344]: time="Dec 12 05:31:32.139" level=debug msg="Submitted blob to DA successfully.[]" module=celestia
Dec 12 05:31:32 ip-10-0-2-51 roller[249344]: time="Dec 12 05:31:32.177" level=info msg="Submitted batch to DA.[start height 13634 end height 13634]" module=block_manager
Dec 12 05:32:25 ip-10-0-2-51 roller[249344]: time="Dec 12 05:32:25.791" level=error msg="Pausing block production until new batch is submitted.[Batch skew time 3m3.795366588s Max batch skew time 2m0s Last block in settlement time 2024-12-12 05:29:21.975840402 +0000 UTC]" module=block_manager
Dec 12 05:32:25 ip-10-0-2-51 roller[249344]: time="Dec 12 05:32:25.791" level=error msg="node is unhealthy[error Block production paused. Time between last block produced and last block submitted higher than max skew time: 2m0s last block in settlement time: 2024-12-12 05:29:21.975840402 +0000 UTC resource exhausted]"
Dec 12 05:33:32 ip-10-0-2-51 roller[249344]: time="Dec 12 05:33:32.795" level=info msg="Submitted batch to SL.[start height 13634 end height 13634]" module=block_manager
Dec 12 05:33:32 ip-10-0-2-51 roller[249344]: time="Dec 12 05:33:32.795" level=debug msg="Submitted a batch to both sub-layers.[n bytes consumed from pending 752 pending after 1505 skew time 2m3.599502386s]" module=block_manager
Dec 12 05:33:36 ip-10-0-2-51 roller[249344]: time="Dec 12 05:33:36.821" level=debug msg="Submitted blob to DA successfully.[]" module=celestia
Dec 12 05:33:36 ip-10-0-2-51 roller[249344]: time="Dec 12 05:33:36.835" level=info msg="Submitted batch to DA.[start height 13635 end height 13636]" module=block_manager

[srene]

batch_acceptance_attempts = "5"
batch_acceptance_timeout = "2m0s"
batch_submit_bytes = 500000
batch_submit_max_time = "1m0s"
batch_submit_time = "1m0s"
block_batch_size = "500"
block_time = "0.2s"
empty_blocks_max_time = "3600s"
max_idle_time = "1m0s"
max_proof_time = "5s"
max_skew_time = "2m0s

[srene]

it may be just a problem related to the config used (very small max skew time and high submission time and block creation time). but need to check

[danwt]

What's the desired behavior @srene ?

[srene]

closing it. it's been already solved in #1252
i think the problem is that it was waiting to submit all pending batches, while only submitting one should be necessary to restart block production.