Prometheus telemetry #1526

rakanalh · 2024-11-26T11:33:37Z

Description

From the referenced issue, the following was the list of metrics to collect:

All

Storage size ( I think this can be fed into prometheus outside the scope of citrea)
RPC served in last 5 min (Same as this one, nginx?)

Sequencer

Current number txs in mempool
Mempool inbound tx/s (Can be calculated from the current number)
dry_run_transactions execution time
produce_block transactions execution time
(this one will be hard to track so if we cant no problem) avg. wait time for tx in mempool (Can be calculated i believe)
Send sequencer commitment execution time
Sequencer commitment number of blocks
Current L1 block number used in block production

Full node

Sync progress (Not applicable since we have to know sequencer latest L2 vs fullnode or prover current L2)
Current L2 block number
Current L1 block number
Soft confirmation pull rate (important while syncing)
Soft confirmaiton process durations
L1 scan duration per L1 block

Prover(s)

Current L2 block number
Current L1 blcok number
Cycle count for each proof
DA tx mining time

TODO

Implement / Export graphana dashboards to display these metrics.

Linked Issues

Fixes Profiling / metrics for full nodes, provers, sequencer #1293

codecov · 2024-12-09T18:13:44Z

Codecov Report

Attention: Patch coverage is 81.85053% with 153 lines in your changes missing coverage. Please review.

Project coverage is 76.2%. Comparing base (8535094) to head (9d72d33).
Report is 3 commits behind head on nightly.

Files with missing lines	Patch %	Lines
crates/common/src/telemetry/mod.rs	0.0%	51 Missing ⚠️
crates/light-client-prover/src/runner.rs	0.0%	23 Missing ⚠️
bin/citrea/src/rollup/mod.rs	66.6%	18 Missing ⚠️
crates/batch-prover/src/runner.rs	51.4%	17 Missing ⚠️
crates/fullnode/src/runner.rs	52.7%	17 Missing ⚠️
crates/sequencer/src/runner.rs	57.5%	17 Missing ⚠️
crates/light-client-prover/src/da_block_handler.rs	0.0%	6 Missing ⚠️
crates/bitcoin-da/src/service.rs	62.5%	3 Missing ⚠️
bin/citrea/src/rollup/bitcoin.rs	0.0%	1 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
bin/citrea/src/main.rs	`0.0% <ø> (ø)`
bin/citrea/src/rollup/mock.rs	`80.0% <100.0%> (ø)`
crates/batch-prover/src/da_block_handler.rs	`78.3% <100.0%> (+0.4%)`	⬆️
crates/common/src/config.rs	`95.1% <100.0%> (+0.3%)`	⬆️
crates/common/src/telemetry/provers.rs	`100.0% <100.0%> (ø)`
crates/fullnode/src/da_block_handler.rs	`77.1% <100.0%> (+0.9%)`	⬆️
crates/fullnode/src/telemetry.rs	`100.0% <100.0%> (ø)`
crates/primitives/src/utils.rs	`100.0% <100.0%> (ø)`
crates/risc0/src/host.rs	`72.2% <100.0%> (+1.6%)`	⬆️
crates/sequencer/src/commitment/mod.rs	`86.8% <100.0%> (+1.1%)`	⬆️
... and 20 more

... and 8 files with indirect coverage changes

crates/batch-prover/src/telemetry.rs

eyusufatik · 2024-12-10T08:54:07Z

(Cycle count is NOT part of the output, cannot be reported)

in risc0 adapter, here you can get cycle counts from stats. change return types if you wish, but we need this information. probably the most important telemetry both provers can have. also why don't you have any telemetry implemented for light client prover 😅?

        let ProveInfo { receipt, stats } =
            prover.prove_with_opts(env, &elf, &ProverOpts::groth16())?;

ercecan

LGTM small fixes and couple of suggestions

crates/batch-prover/src/telemetry.rs

crates/sovereign-sdk/rollup-interface/src/node/telemetry.rs

crates/common/src/telemetry.rs

crates/fullnode/src/telemetry.rs

crates/primitives/src/utils.rs

crates/sequencer/src/runner.rs

crates/sovereign-sdk/full-node/db/sov-schema-db/src/iterator.rs

rakanalh · 2024-12-10T10:52:53Z

(Cycle count is NOT part of the output, cannot be reported)

in risc0 adapter, here you can get cycle counts from stats. change return types if you wish, but we need this information. probably the most important telemetry both provers can have. also why don't you have any telemetry implemented for light client prover 😅?
        let ProveInfo { receipt, stats } =
            prover.prove_with_opts(env, &elf, &ProverOpts::groth16())?;

Added... thanks for the hint.

yaziciahmet

Good PR. Left some comments as in it's current state. I will re-review once PR is not draft anymore.

yaziciahmet · 2024-12-10T12:43:33Z

resources/configs/bitcoin-regtest/batch_prover_rollup_config.toml

can you make telemetry ports of sequencer, fullnode, and prover for mock and regtest different? it causes problems when running different nodes locally.

yaziciahmet · 2024-12-10T12:45:59Z

guests/risc0/light-client-proof-mock/Cargo.lock

@@ -716,6 +716,7 @@ dependencies = [
 "bincode",
 "borsh",
 "hex",
+ "prometheus-client",


it would be great if you can guard this dep behind native feature.

yaziciahmet · 2024-12-10T12:48:20Z

docker/docker-compose.telemetry.yaml

With this, do we have to spawn up a prometheus container for working locally?

yaziciahmet · 2024-12-10T12:50:56Z

crates/sovereign-sdk/rollup-interface/src/node/services/da.rs

@@ -127,6 +129,9 @@ pub trait DaService: Send + Sync + 'static {
        &self,
        sequencer_da_pub_key: &[u8],
    ) -> Vec<SequencerCommitment>;
+
+    /// Returns the telemetry targets type for the DA service.
+    fn telemetry_targets(&self) -> &DaTelemetryTargets;


This should be native only as well

yaziciahmet · 2024-12-10T13:11:30Z

crates/sovereign-sdk/full-node/db/sov-schema-db/src/lib.rs

+#[inline]
+pub fn duration_to_seconds(d: Duration) -> f64 {
+    let nanos = f64::from(d.subsec_nanos()) / 1e9;
+    d.as_secs() as f64 + nanos
+}


IMO nanos is a bit too much precision. We can use micros and just do d.as_micros() as u64 with no overflow no floating point whatsoever.

yaziciahmet · 2024-12-10T13:22:47Z

crates/sovereign-sdk/full-node/db/sov-schema-db/src/iterator.rs

+            .get_or_create(&vec![(
+                "cf_name".to_owned(),
+                S::COLUMN_FAMILY_NAME.to_owned(),
+            )])


get_or_create is used everywhere in sov-schema-db, but we already know the dbs, and column families. can we somehow create these at the startup as well?

rakanalh · 2024-12-11T13:17:35Z

Closing in favor of #1589 1589

rakanalh added 2 commits December 3, 2024 16:38

Initial setup

7281f11

Start telemetry server

3d6a341

rakanalh force-pushed the rakanalh/prometheus branch from cf570fb to 3d6a341 Compare December 3, 2024 13:38

rakanalh added 16 commits December 4, 2024 13:50

Setup local prometheus / graphana

e86cc67

Migrate sov-schema-db metrics

f399c78

Add more metrics to sequencer

169888d

Refine spawning server

a891f3e

Track mempool transactions

1a2ad12

Merge remote-tracking branch 'origin/nightly' into rakanalh/prometheus

7f710e2

Complete sequencer telemetry target implementation

6b573ac

Add utils file

dff72ce

Add telemetry config

0cb7060

USe telemetry config to start server

5b3ba12

Setup telemetry for Fullnode

aa17800

Implement fullnode telemetry

79144de

Merge remote-tracking branch 'origin/nightly' into rakanalh/prometheus

6e31149

Prover telemetry

7454334

Include schema-db telemetry

8a4681a

Restore guest Cargo.lock

439a6a0

rakanalh marked this pull request as ready for review December 9, 2024 13:07

auto-assign bot requested review from ercecan, eyusufatik, jfldde, kpp and yaziciahmet December 9, 2024 13:07

auto-assign bot assigned rakanalh Dec 9, 2024

rakanalh added 4 commits December 9, 2024 16:35

Fix test compilation

6272395

Implement proper default for Telemetry config

a21ddc3

Fix clippy

114f7d5

Remove citrea-primitives from sov-rollup interface

affd6cd

rakanalh added 2 commits December 9, 2024 19:07

Add missing config

c06bcd7

Fix env test

800e22e

rakanalh force-pushed the rakanalh/prometheus branch from 13a37ac to 800e22e Compare December 9, 2024 16:12

rakanalh added 5 commits December 9, 2024 19:15

Update config files

7e066e6

Merge branch 'nightly' into rakanalh/prometheus

bbff505

Fix doc

247de52

Use citrea-e2e from my branch

d8fd3de

Fix citrea-e2e tests

14f7c10

ercecan reviewed Dec 10, 2024

View reviewed changes

crates/batch-prover/src/telemetry.rs Outdated Show resolved Hide resolved

ercecan requested changes Dec 10, 2024

View reviewed changes

rakanalh added 4 commits December 10, 2024 12:38

Merge remote-tracking branch 'origin/nightly' into rakanalh/prometheus

53e1f46

Record cycle count

4606a4d

Set mempool by len

48f2d11

Fix typo

9f927d1

Add light client telemetry

9d72d33

rakanalh marked this pull request as draft December 10, 2024 11:56

yaziciahmet requested changes Dec 10, 2024

View reviewed changes

rakanalh closed this Dec 11, 2024

rakanalh deleted the rakanalh/prometheus branch December 11, 2024 13:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Prometheus telemetry #1526

Prometheus telemetry #1526

rakanalh commented Nov 26, 2024 •

edited

Loading

codecov bot commented Dec 9, 2024 •

edited

Loading

eyusufatik commented Dec 10, 2024

ercecan left a comment

rakanalh commented Dec 10, 2024

yaziciahmet left a comment

yaziciahmet Dec 10, 2024

yaziciahmet Dec 10, 2024

yaziciahmet Dec 10, 2024 •

edited

Loading

yaziciahmet Dec 10, 2024

yaziciahmet Dec 10, 2024

yaziciahmet Dec 10, 2024

rakanalh commented Dec 11, 2024

Prometheus telemetry #1526

Prometheus telemetry #1526

Conversation

rakanalh commented Nov 26, 2024 • edited Loading

Description

All

Sequencer

Full node

Prover(s)

TODO

Linked Issues

codecov bot commented Dec 9, 2024 • edited Loading

Codecov Report

eyusufatik commented Dec 10, 2024

ercecan left a comment

Choose a reason for hiding this comment

rakanalh commented Dec 10, 2024

yaziciahmet left a comment

Choose a reason for hiding this comment

yaziciahmet Dec 10, 2024

Choose a reason for hiding this comment

yaziciahmet Dec 10, 2024

Choose a reason for hiding this comment

yaziciahmet Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

yaziciahmet Dec 10, 2024

Choose a reason for hiding this comment

yaziciahmet Dec 10, 2024

Choose a reason for hiding this comment

yaziciahmet Dec 10, 2024

Choose a reason for hiding this comment

rakanalh commented Dec 11, 2024

rakanalh commented Nov 26, 2024 •

edited

Loading

codecov bot commented Dec 9, 2024 •

edited

Loading

yaziciahmet Dec 10, 2024 •

edited

Loading