fix: day 1 patch, add dashboards #3927

MasterPtato · 2026-01-16T00:14:50Z

No description provided.

railway-app · 2026-01-16T00:14:58Z

🚅 Deployed to the rivet-pr-3927 environment in rivet-frontend

Service	Status	Web	Updated (UTC)
website	😴 Sleeping (View Logs)	Web	Jan 16, 2026 at 2:39 am
frontend-inspector	😴 Sleeping (View Logs)	Web	Jan 16, 2026 at 2:39 am
frontend-cloud	😴 Sleeping (View Logs)	Web	Jan 16, 2026 at 2:38 am

MasterPtato · 2026-01-16T00:15:15Z

feat: add actor and kv metrics #3928
fix: day 1 patch, add dashboards #3927 👈 (View in Graphite)
main

How to use the Graphite Merge Queue

Add the label merge-queue to this PR to add it to the merge queue.

You must have a Graphite account in order to use the merge queue. Sign up using this link.

_{An organization admin has enabled the Graphite Merge Queue in this repository.} _{Please do not merge from GitHub as this will restart CI on PRs being processed by the merge queue.}

This stack of pull requests is managed by Graphite. Learn more about stacking.

claude · 2026-01-16T00:16:30Z

Pull Request Review: fix: day 1 patch, add dashboards

Summary

This PR includes a collection of bug fixes and adds two new Grafana dashboards (Epoxy and Pegboard). The changes are mostly good, with a few minor issues and observations.

Code Quality & Best Practices

✅ Good Patterns Observed

Import organization (api-public/src/actors/kv_get.rs:5): Moving the Json import to the top-level imports is cleaner and follows the codebase convention of keeping all imports at the top of the file.
Metrics consistency (guard/src/routing/mod.rs:130,147): Fixing metric label casing from "GATEWAY" and "API" to lowercase "gateway" and "api" improves consistency across metrics.
Dashboard additions: New monitoring dashboards for Epoxy and Pegboard will improve observability.

Potential Issues & Concerns

⚠️ Code Issues

Missing metric decrement before early returns (pegboard-runner/src/lib.rs:252-258)
- Location: The CONNECTION_ACTIVE.dec() metric is only decremented at the end of the function (line 252), but there are several early error returns that bypass this code.
- Problem: If update_alloc_res fails or other errors occur before line 252, the metric will never be decremented, leading to incorrect active connection counts.
- Fix: The metric increment at line 138-144 should be paired with a decrement that is guaranteed to run (e.g., using a RAII guard pattern or ensuring it runs in all code paths).
- Severity: Medium - This will cause metric drift over time, making monitoring unreliable.
Unused parameter warning (gasoline/src/db/kv/mod.rs:623)
- Issue: The parameter name changed from _worker_id to worker_id, but the diff doesn't show where this parameter is actually being used.
- Question: Is worker_id actually used in the function body, or should it remain prefixed with underscore to avoid compiler warnings?
- Severity: Minor - May cause compiler warnings if unused.
Breaking change to metric name (gasoline/src/db/kv/mod.rs:226,1010)
- Issue: SignalPending is renamed to SignalPending2.
- Concern: This is a breaking change for any existing Grafana dashboards, alerts, or monitoring tools that reference the old metric name.
- Question: Is this intentional? If so, consider:
  - Documenting the migration path
  - Updating any dependent dashboards
  - Consider keeping both metrics temporarily for backwards compatibility
- Severity: Medium - Could break existing monitoring infrastructure.

Performance Considerations

No significant performance issues identified. The changes are mostly small fixes and additions.

Security Concerns

No security issues identified in this PR.

Test Coverage

⚠️ No test changes included: This PR makes functional changes but doesn't include any test updates or additions. Consider:

Testing the metric increment/decrement logic in pegboard-runner
Testing the renamed metric in gasoline
Integration tests for the routing label fixes

Recommendations

High Priority

Fix metric decrement in pegboard-runner: Ensure CONNECTION_ACTIVE is decremented in all code paths, not just the happy path.

Medium Priority

Clarify SignalPending2 rename: Document why this breaking change was made and update any dependent systems.
Review worker_id usage: Verify the parameter is actually used, or revert to _worker_id.

Low Priority

Add tests: Include test coverage for the behavioral changes.
Grafana dashboard updates: Ensure existing dashboards are updated for the SignalPending2 metric name change.

Files Reviewed

✅ engine/packages/api-public/src/actors/kv_get.rs
✅ engine/packages/gasoline/src/db/kv/mod.rs
✅ engine/packages/gasoline/src/metrics.rs
✅ engine/packages/guard/src/routing/mod.rs
✅ engine/packages/pegboard-runner/src/lib.rs
📊 engine/docker/template/grafana-dashboards/epoxy.json
📊 engine/docker/template/grafana-dashboards/pegboard.json
📊 engine/docker/template/grafana-dashboards/gasoline.json (updated)
📊 engine/docker/template/grafana-dashboards/guard.json (updated)

Overall Assessment

This is a reasonable collection of bug fixes and monitoring improvements. The main concern is the metric decrement issue in pegboard-runner that could lead to incorrect metrics. The SignalPending2 rename also needs clarification to ensure it doesn't break existing monitoring.

Recommendation: Address the metric decrement issue before merging, and clarify/document the SignalPending2 change.

pkg-pr-new · 2026-01-16T00:16:56Z

More templates

@rivetkit/cloudflare-workers

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/cloudflare-workers@3927

@rivetkit/db

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/db@3927

@rivetkit/framework-base

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/framework-base@3927

@rivetkit/next-js

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/next-js@3927

@rivetkit/react

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/react@3927

rivetkit

pnpm add https://pkg.pr.new/rivet-dev/rivet/rivetkit@3927

@rivetkit/sql-loader

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/sql-loader@3927

@rivetkit/virtual-websocket

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/virtual-websocket@3927

@rivetkit/engine-runner

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner@3927

@rivetkit/engine-runner-protocol

pnpm add https://pkg.pr.new/rivet-dev/rivet/@rivetkit/engine-runner-protocol@3927

commit: ee27a16

graphite-app · 2026-01-16T02:34:26Z

engine/docker/template/grafana-dashboards/epoxy.json

+					"expr": "sum by (rivet_datacenter) (rate(rivet_epoxy_proposals_total{rivet_project=~\"$project\",rivet_datacenter=~\"$datacenter\"}[$__rate_interval]))",
+					"instant": false,
+					"legendFormat": "{{workflow_name}}",


Query groups by rivet_datacenter but legendFormat references {{workflow_name}} which doesn't exist in the query result. This will cause the legend to display incorrectly or be empty.

Fix:

"legendFormat": "{{rivet_datacenter}}"

Spotted by Graphite Agent

Is this helpful? React 👍 or 👎 to let us know.

graphite-app · 2026-01-16T02:34:27Z

engine/docker/template/grafana-dashboards/epoxy.json

+					"expr": "sum by (rivet_datacenter) (rate(rivet_epoxy_pre_accept_total{rivet_project=~\"$project\",rivet_datacenter=~\"$datacenter\"}[$__rate_interval]))",
+					"instant": false,
+					"legendFormat": "{{workflow_name}}",


Query groups by rivet_datacenter but legendFormat uses {{workflow_name}}. The label in the legend template must match the grouping field.

Fix:

"legendFormat": "{{rivet_datacenter}}"

Spotted by Graphite Agent

Is this helpful? React 👍 or 👎 to let us know.

graphite-app · 2026-01-16T02:34:29Z

engine/docker/template/grafana-dashboards/epoxy.json

+					"expr": "sum by (rivet_datacenter) (rate(rivet_epoxy_accept_total{rivet_project=~\"$project\",rivet_datacenter=~\"$datacenter\"}[$__rate_interval]))",
+					"instant": false,
+					"legendFormat": "{{workflow_name}}",


Mismatch between grouping (rivet_datacenter) and legendFormat ({{workflow_name}}). This pattern repeats throughout the dashboard and will result in incorrect or missing legend labels.

Fix:

"legendFormat": "{{rivet_datacenter}}"

Spotted by Graphite Agent

Is this helpful? React 👍 or 👎 to let us know.

MasterPtato requested a review from NathanFlurry January 16, 2026 00:14

railway-app bot temporarily deployed to rivet-frontend / rivet-pr-3927 January 16, 2026 00:15 Destroyed

fix: day 1 patch, add dashboards

ee27a16

MasterPtato force-pushed the 01-15-fix_day_1_patch_add_dashboards branch from f7e6088 to ee27a16 Compare January 16, 2026 02:31

railway-app bot deployed to rivet-frontend / rivet-pr-3927 January 16, 2026 02:31 View deployment

MasterPtato mentioned this pull request Jan 16, 2026

feat: add actor and kv metrics #3928

Draft

graphite-app bot reviewed Jan 16, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: day 1 patch, add dashboards #3927

fix: day 1 patch, add dashboards #3927

Uh oh!

MasterPtato commented Jan 16, 2026

Uh oh!

railway-app bot commented Jan 16, 2026 •

edited

Loading

Uh oh!

MasterPtato commented Jan 16, 2026 •

edited

Loading

Uh oh!

claude bot commented Jan 16, 2026 •

edited

Loading

Uh oh!

pkg-pr-new bot commented Jan 16, 2026 •

edited

Loading

Uh oh!

graphite-app bot Jan 16, 2026

Uh oh!

graphite-app bot Jan 16, 2026

Uh oh!

graphite-app bot Jan 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

fix: day 1 patch, add dashboards #3927

Are you sure you want to change the base?

fix: day 1 patch, add dashboards #3927

Uh oh!

Conversation

MasterPtato commented Jan 16, 2026

Uh oh!

railway-app bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MasterPtato commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How to use the Graphite Merge Queue

Uh oh!

claude bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pull Request Review: fix: day 1 patch, add dashboards

Summary

Code Quality & Best Practices

✅ Good Patterns Observed

Potential Issues & Concerns

⚠️ Code Issues

Performance Considerations

Security Concerns

Test Coverage

Recommendations

High Priority

Medium Priority

Low Priority

Files Reviewed

Overall Assessment

Uh oh!

pkg-pr-new bot commented Jan 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

graphite-app bot Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

graphite-app bot Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

graphite-app bot Jan 16, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

railway-app bot commented Jan 16, 2026 •

edited

Loading

MasterPtato commented Jan 16, 2026 •

edited

Loading

claude bot commented Jan 16, 2026 •

edited

Loading

pkg-pr-new bot commented Jan 16, 2026 •

edited

Loading