feat(synth-bench): automate TPS measurement for wait_until: NONE
#12874
Conversation
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## master #12874 +/- ##
==========================================
- Coverage 70.54% 70.52% -0.02%
==========================================
Files 851 851
Lines 174983 174983
Branches 174983 174983
==========================================
- Hits 123433 123407 -26
- Misses 46419 46439 +20
- Partials 5131 5137 +6
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
benchmarks/synth-bm/src/metrics.rs
Outdated
| impl MetricName { | ||
| fn report_name(self) -> &'static str { | ||
| match self { | ||
| MetricName::SuccessfulTransactions => "near_transaction_processed_successfully_total", |
There was a problem hiding this comment.
One important thing to keep in mind is that this metric is incremented when the transaction successfully converts into a receipt. It does not mean that the receipt is actually getting executed (it might even happen in a different block if the receipt gets delayed.) You will probably want a metric along the lines of near_action_called_count (allows determining the rate of actions based on action types even in mixed action workloads, or where receipts have multiple actions!) or receipts (I don't think we have a single metric for all of them together, but shouldn't be too hard to add up the delayed, local, incoming, etc. receipts.)
There was a problem hiding this comment.
Interesting! We've been using near_transaction_processed_successfully_total for that purpose since the ft benchmarking days...
Will look into near_action_called_count.
There was a problem hiding this comment.
AFAIK this is currently the state of the art for determining successful txs in benchmarks. Usage of synth-bm is increasing, so I'd like to propose fixing this separately in order to merge this PR soon. Opened #12931.
There was a problem hiding this comment.
Even if transaction_processed doesn't measure what it was expected to, it might still be useful to have support for that metric in the tooling. E.g. to check if all txs could successfully be converted to receipts.
There was a problem hiding this comment.
I'd be interested to know if transaction_processed is the best way to have right now to measure throughput , or if we should use / implement something else. Just asking because I've been using transaction_processed in my single and multi node measurements (only simple transfers , no complex operations)
There was a problem hiding this comment.
I think in your setup you're also getting numbers from /metrics and native transfers are a single action. So you could get the number of successful native transfers from the line in the /metrics response that looks like:
near_action_called_count{action="Transfer"} 200
(The line may be missing when no transfers were processed successfully yet.)
| @@ -84,6 +85,11 @@ pub async fn benchmark(args: &BenchmarkArgs) -> anyhow::Result<()> { | |||
| rpc_response_handler.handle_all_responses().await; | |||
| }); | |||
|
|
|||
| let transaction_stat_service = | |||
There was a problem hiding this comment.
I have a request: enable (or disable) transaction_stat_service through a command line flag.
The reason is that in my current workloads I don't always need this functionality. I could ignore the output, but I think it's cleaner to have a flag to turn this off
As described in #12806, the RPC response to transactions sent with
wait_until: NONEdoesn't tell much. In that case, measuring TPS in terms of successfully processed transactions was a manual process. This PR automates it and uses that functionality for benchmarking native transfers.The
TransactionsStatisticsServiceadded here can in the future also be used in other workloads.Testing
/metricsendpoint.just benchmark_native_transfersprints TPS of successfully processed transactions.Closes #12806