Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pending submissions metrics don't update when block production paused #1221

Closed
Northa opened this issue Nov 11, 2024 · 0 comments · Fixed by #1313
Closed

Pending submissions metrics don't update when block production paused #1221

Northa opened this issue Nov 11, 2024 · 0 comments · Fixed by #1313
Assignees

Comments

@Northa
Copy link

Northa commented Nov 11, 2024

dymint/block/submit.go

Lines 80 to 85 in ed72d9b

types.RollappPendingSubmissionsSkewBytes.Set(float64(pendingBytes.Load()))
types.RollappPendingSubmissionsSkewBlocks.Set(float64(unsubmittedBlocks()))
submitter.Nudge()
}
})

During some internal tests i noticed that rollapp_pending_submissions_skew_batches metric not updating in real time.

Steps to reproduce.

  1. a working rollapp setup with da-client.
  2. restart\stop hub rpc.
  3. Simulate a load test by sending some txs into rollapp.
  4. Monitor the rollapp_pending_submissions_skew_batches metric.

possible fix: Northa@9c68066

UPDATE2:
When it comes to pause block production due to too many batches are pending submission metrics are not updating as well.
Affected metrics

rollapp_pending_submissions_skew_bytes
rollapp_pending_submissions_skew_batches
rollapp_pending_submissions_skew_blocks

time="Nov 12 13:45:31.686" level=error msg="Enough bytes to build a batch have been accumulated, but too many batches are pending submission. Pausing block production until a signal is consumed.[]" module=block_manager

Not sure but probably metrics must be updated there

dymint/block/submit.go

Lines 129 to 131 in 30a84df

ticker.Reset(maxBatchTime)
pending = uatomic.Uint64Sub(&pendingBytes, nConsumed)
logger.Info("Submitted a batch to both sub-layers.", "n bytes consumed from pending", nConsumed, "pending after", pending) // TODO: debug level

@danwt danwt changed the title Update RollappPendingSubmissionsSkewBatches each block Pending submissions metrics don't update when block production paused due to 'too many batches are pending submission' Jan 7, 2025
@danwt danwt changed the title Pending submissions metrics don't update when block production paused due to 'too many batches are pending submission' Pending submissions metrics don't update when block production paused due Jan 7, 2025
@danwt danwt changed the title Pending submissions metrics don't update when block production paused due Pending submissions metrics don't update when block production paused Jan 7, 2025
@mtsitrin mtsitrin linked a pull request Jan 8, 2025 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants