cuprated: RPC handlers #355

hinto-janai · 2024-12-05T01:45:06Z

What

Implements the {json-rpc, binary, json} handlers in cuprated; adds various types and changes some misc things as needed.

How

See below review.

miner transaction was not accounted for

hinto-janai

@Boog900 I think this mostly needs review on:

general direction and broader changes
viability of new tower::Service request/response types

I don't think you have to review each handler fn individually in-depth since there will eventually be the testing harness that proves input/output matches monerod. With that said, here's some info:

RPC code structure

Directory	Contains
`binaries/cuprated/src/rpc/handlers/`	The RPC handler functions, shared functions, and helper functions
`binaries/cuprated/src/rpc/service/`	`fn` versions of our `tower::Service`s to reduce noise

RPC handlers

Color	Meaning
🟢	Ready
🟡	Ready but could be more efficient
🟣	Ready but callstack depends on other things
🟠	Depends on other things
🔴	Waiting on binary strings
🔵	Unsupported (for now)
⚪	Unsupported (probably forever)
⚫	I think these could be deprecated or have different behavior

JSON-RPC `fn`	Status	Details
get_block_count	🟢
get_last_block_header	🟢
get_block_header_by_hash	🟢
get_block_header_by_height	🟢
get_block	🟢
hard_fork_info	🟢
on_get_block_hash	🟢
get_block_headers_range	🟡	Would benefit from a request that allows retrieving a range of `(Block, ExtendedBlockHeader)`
get_connections	🟣	Waiting on address book `Service` impl
set_bans	🟣	^
get_bans	🟣	^
banned	🟣	^
get_version	🟣	Waiting on blockchain context `Service` impl
get_output_histogram	🟣	^
get_fee_estimate	🟣	^
calc_pow	🟣	^
flush_transaction_pool	🟣	Waiting on txpool manager `Service` impl
relay_tx	🟣	^
get_coinbase_tx_sum	🟣	Waiting on blockchain `Service` impl
get_alternate_chains	🟣	^
sync_info	🟣	Waiting on multiple `Service` impls
get_miner_data	🟣	Waiting on txpool `Service` impl
submit_block	🟣	^
get_info	🟣	Needs access to unimplemented things from various places
generate_blocks	🟣	Needs access to `cuprated`'s `Chain` and `--regtest`
add_aux_pow	🟣	Waiting on crypto functions
get_transaction_pool_backlog	🔴
get_output_distribution	🔴
get_tx_ids_loose	🔵	Not implemented in `monerod` release branch yet
flush_cache	⚪	`cuprated` does not need this
prune_blockchain	⚫	I don't think an always available RPC method is necessary for something that is done once. This could return if the chain is pruned + the pruning seed but not actually prune if that makes things more complex. Pruning itself should be done with the equivalent of `--prune-blockchain`.

Other JSON `fn`	Status	Details
get_height	🟢
get_outs	🟢
is_key_image_spent	🟣	Waiting on txpool `Service` impl
get_transaction_pool_hashes	🟣	^
get_transaction_pool	🟣	^
get_transaction_pool_stats	🟣	^
save_bc	🟣	Waiting on blockchain manager `Service` impl
stop_daemon	🟣	^
pop_blocks	🟣	^
get_peer_list	🟣	Waiting on address book `Service` impl
get_public_nodes	🟣	^
get_alt_blocks_hashes	🟣	Waiting on blockchain `Service` impl
send_raw_transaction	🟣	Waiting on txpool manager `Service` impl
get_transactions	🟣	Waiting on JSON representation of `Transaction`
get_limit	🟠	Waiting on P2P interface
set_limit	🟠	^
out_peers	🟠	^
in_peers	🟠	^
get_net_stats	🟠	^
set_log_level	🔵	Will use `tracing` levels
set_log_categories	🔵	Could be re-purposed to use `tracing` filters
set_bootstrap_daemon	🔵	Needs bootstrap implementation
start_mining	⚪	`cuprated` does not mine
stop_mining	⚪	^
mining_status	⚪	^
set_log_hash_rate	⚪	^
update	⚫	This could return if an update is available and related info but not actually self-update. Software upgrades should be done by the user and/or a package manager.

Binary `fn`	Status	Details
get_blocks_by_height	🟢
get_hashes	🟢
get_output_indexes	🟢
get_outs	🟢
get_blocks	🟣	Waiting on txpool `Service` impl
get_transaction_pool_hashes	🟣	^
get_output_distribution	🔴	Although this is binary, the internal fn is shared with JSON-RPC's `get_output_distribution`

hinto-janai · 2024-12-21T01:09:59Z

Cargo.toml

+	# Types
+	"types/types",
+	"types/hex",
+	"types/fixed-bytes",


cuprate_types::hex::Hex was moved into a separate crate cuprate_hex such that the types could implement epee without running into cyclic dependency or orphan impl problems.

I moved fixed-bytes as well since that is used in multiple places outside of networking.

hinto-janai · 2024-12-25T19:40:04Z

rpc/types/src/macros.rs

@@ -291,6 +291,7 @@ macro_rules! define_response {
        }
    ) => {
        $( #[$attr] )*
+        #[cfg_attr(feature = "serde", serde(default))] // TODO: link epee field not serializing oddity


For the time being, I'm making response types in cuprate-rpc-types have serde(default) since:

monerod will not serialize fields with empty containers

monerod's response types sometimes change depending on branches

serde(default) is a bit of a blunt tool to use but I think it works for now. I will think about this more when integrating RPC into cuprated and with more tests.

hinto-janai · 2024-12-25T19:42:50Z

types/types/src/rpc/types.rs

-//! These are `struct`s that appear in request/response types.
-//! For example, [`crate::json::GetConnectionsResponse`] contains
-//! the [`crate::misc::ConnectionInfo`] struct defined here.
+//! Various types (in)directly used in RPC.


1/2

This is a new module, cuprate_types::rpc. It contains types that are exposed by RPC but are also used internally.

The difference between cuprate_rpc_types and cuprate_types::rpc is that the latter has more canonical types where it makes sense, e.g. [u8; 32] instead of String.

hinto-janai · 2024-12-25T19:44:49Z

rpc/types/src/from.rs

@@ -0,0 +1,190 @@
+//! [`From`] implementations from other crate's types into [`crate`] types.


2/2

Since cuprate_types::rpc needs to map to cuprate_rpc_types eventually, instead of doing it adhoc-ly in places, this file defines a bunch of

impl From<cuprate_types::rpc::*> for cuprate_rpc_types::*

plus some other types from other crates.

hinto-janai · 2024-12-25T19:52:34Z

rpc/types/src/json.rs

-    #[doc = serde_doc_test!(
-        GET_BLOCK_TEMPLATE_RESPONSE => GetBlockTemplateResponse {
-            base: ResponseBase::OK,
-            blockhashing_blob: "1010f4bae0b4069d648e741d85ca0e7acb4501f051b27e9b107d3cd7a3f03aa7f776089117c81a00000000e0c20372be23d356347091025c5b5e8f2abf83ab618378565cce2b703491523401".into(),
-            blocktemplate_blob: "1010f4bae0b4069d648e741d85ca0e7acb4501f051b27e9b107d3cd7a3f03aa7f776089117c81a0000000002c681c30101ff8a81c3010180e0a596bb11033b7eedf47baf878f3490cb20b696079c34bd017fe59b0d070e74d73ffabc4bb0e05f011decb630f3148d0163b3bd39690dde4078e4cfb69fecf020d6278a27bad10c58023c0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000".into(),
-            difficulty_top64: 0,
-            difficulty: 283305047039,
-            expected_reward: 600000000000,
-            height: 3195018,
-            next_seed_hash: "".into(),
-            prev_hash: "9d648e741d85ca0e7acb4501f051b27e9b107d3cd7a3f03aa7f776089117c81a".into(),
-            reserved_offset: 131,
-            seed_hash: "e2aa0b7b55042cd48b02e395d78fa66a29815ccc1584e38db2d1f0e8485cd44f".into(),
-            seed_height: 3194880,
-            wide_difficulty: "0x41f64bf3ff".into(),
-        }
-    )]


I think doing tests like this was a bad tradeoff, it's a bit too noisy and IDE stuff stops working, so a large majority of the changes to rpc/types/src/{json,bin,other}.rs are moving the macro tests into normal #[test] fn() tests at the bottom of the file.

hinto-janai · 2024-12-25T19:54:16Z

rpc/types/src/json.rs

-        blockhashing_blob: String,
-        blocktemplate_blob: String,
+        blockhashing_blob: HexVec,
+        blocktemplate_blob: HexVec,
        difficulty_top64: u64,
        difficulty: u64,
        expected_reward: u64,
        height: u64,
-        next_seed_hash: String,
-        prev_hash: String,
+        /// This is a [`Hex<32>`] that is sometimes empty.
+        next_seed_hash: HexVec,
+        prev_hash: Hex<32>,
        reserved_offset: u64,
-        seed_hash: String,
+        seed_hash: Hex<32>,


cuprate-rpc-types now uses Hex and HexVec over String where it can.

hinto-janai · 2024-12-25T19:55:54Z

binaries/cuprated/src/rpc/handlers/json_rpc.rs

+    state: CupratedRpcHandler,
+    _: GetConnectionsRequest,
+) -> Result<GetConnectionsResponse, Error> {
+    let connections = address_book::connection_info::<ClearNet>(&mut DummyAddressBook).await?;


All RPC calls involving the address book service are hardcoded to ClearNet for now. Can be (easily?) changed later.

hinto-janai · 2024-12-25T21:27:13Z

helper/src/fmt.rs

+//! String formatting.
+
+/// A type that can be represented in hexadecimal (with a `0x` prefix).
+pub trait HexPrefix {
+    /// Turn `self` into a hexadecimal string prefixed with `0x`.
+    fn hex_prefix(self) -> String;
+}


Lots of fields in RPC are String where it is a hexadecimal encoded u128 with a lowercase prefix, e.g. 0x0

You can use format!("{var:#x}") for this although it's a bit obscure so I made this trait so you can do var.hex_prefix(). Could remove this if wanted.

hinto-janai · 2024-12-25T21:37:53Z

storage/blockchain/src/ops/output.rs

+    let txid = {
+        let height = u32_to_usize(output.height);
+        let tx_idx = u64_to_usize(output.tx_idx);
+        if let Some(hash) = table_block_txs_hashes.get(&height)?.get(tx_idx) {
+            *hash
+        } else {
+            let miner_tx_id = table_block_infos.get(&height)?.mining_tx_index;
+            let tx_blob = table_tx_blobs.get(&miner_tx_id)?;
+            Transaction::read(&mut tx_blob.0.as_slice())?.hash()
+        }
+    };
+
    Ok(OutputOnChain {
        height: output.height as usize,
        time_lock,
        key,
        commitment,
+        txid,
    })


This function now returns the txid for each output due to https://www.getmonero.org/resources/developer-guides/daemon-rpc.html#get_outs.

This is more expensive, the txid is not required by default as well. Should this be something like txid: Option<[u8; 32]> instead?

hinto-janai · 2024-12-25T21:43:27Z

storage/blockchain/src/service/read.rs

-fn key_images_spent(env: &ConcreteEnv, key_images: HashSet<KeyImage>) -> ResponseResult {
+fn key_images_spent(env: &ConcreteEnv, key_images: Vec<KeyImage>) -> ResponseResult {


This was changed to Vec due to https://www.getmonero.org/resources/developer-guides/daemon-rpc.html#is_key_image_spent behavior.

curl http://127.0.0.1:18081/is_key_image_spent -d '{"key_images":["8d1bd8181bf7d857bdb281e0153d84cd55a3fcaa57c3e570f4a49f935850b5e3","8d1bd8181bf7d857bdb281e0153d84cd55a3fcaa57c3e570f4a49f935850b5e3"]}' -H 'Content-Type: application/json'

{ "credits": 0, "spent_status": [1,1], "status": "OK", "top_hash": "", "untrusted": false }

If cuprated used HashSet then it would respond with something like "spent_status": [1]. Small detail but I think it could matter with wallets due to indexing code depending on this.

hinto-janai added 30 commits October 9, 2024 20:47

import diffs

d170b46

small fixes, hardfork changes

02e950b

lints

c18ba6e

hard_fork

93be3be

apply diffs

c590bbb

review fixes

cec746b

Merge branch 'hf' into json

e57c887

Merge branch 'main' into json

ac6360a

binaries/cuprated/src/rpc/request: pub(super) -> pub(crate)

2522bb8

add BlockChainContextService, on_get_block_hash

fd136b2

Merge branch 'main' into json

9459d59

map tower::BoxError to anyhow::Error

b82ff27

get_block

8a3229b

connection_info

9422c07

hard_fork_info

50bd576

set_bans

f677b1a

get_bans

37bcbb5

banned

c7ae795

flush_transaction_pool

6af977f

get_output_histogram

1a76934

get_coinbase_tx_sum

1d435cc

get_version

d7fef15

get_fee_estimate

1e0ab56

get_alternate_chains

294df3f

relay_tx

08b5ff2

response_base: fn -> const

2daf255

get_transaction_pool_backlog

a10ed59

prune

d7d8b35

calc_pow

5926f24

add_aux_pow

7148ba1

hinto-janai added 2 commits December 13, 2024 19:30

/send_raw_transaction, /save_bc, response_base

b3314d4

/peerlist

c924f24

github-actions bot added the A-p2p Related to P2P. label Dec 14, 2024

hinto-janai added 23 commits December 13, 2024 20:14

/get_transaction_pool

c5abf9b

/get_transaction_pool_stats

805f475

finish other draft

0b3ee2d

get_blocks_by_height, shared::get_outs

541302e

/get_o_indexes.bin

8cd319c

/get_output_distribution.bin

3664e2f

clippy

f7f9063

/get_blocks.bin

04cdc72

rpc-interface: add restricted invariant comments

3763dc9

restricted json-rpc error

ce6838b

get_output_distribution

aca2855

module cleanup

21e42c1

txpool: all_hashes

ed8de29

HexVec

cb6fdb7

fix get_txid for /get_outs

dff77e8

miner transaction was not accounted for

fix doc tests

d47f8ec

fix conflict

114a5db

json-rpc fixes

b2b7e93

get_transaction_pool_hashes fix

d41014b

rpc/interface: fix cargo hack

ecf5d66

review fixes

85ebc67

Merge branch 'main' into other

04ade52

cargo hack fix

c56ecd3

hinto-janai mentioned this pull request Dec 25, 2024

cuprated: RPC handlers (temporary draft) #272

Closed

hinto-janai commented Dec 25, 2024

View reviewed changes

hinto-janai marked this pull request as ready for review December 25, 2024 23:51

hinto-janai requested a review from Boog900 December 25, 2024 23:51

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cuprated: RPC handlers #355

cuprated: RPC handlers #355

hinto-janai commented Dec 5, 2024 •

edited

Loading

hinto-janai left a comment

hinto-janai Dec 21, 2024

hinto-janai Dec 25, 2024

hinto-janai Dec 25, 2024

hinto-janai Dec 25, 2024

hinto-janai Dec 25, 2024

hinto-janai Dec 25, 2024

hinto-janai Dec 25, 2024

hinto-janai Dec 25, 2024

hinto-janai Dec 25, 2024

hinto-janai Dec 25, 2024

		@@ -0,0 +1,190 @@
		//! [`From`] implementations from other crate's types into [`crate`] types.

		fn key_images_spent(env: &ConcreteEnv, key_images: HashSet<KeyImage>) -> ResponseResult {
		fn key_images_spent(env: &ConcreteEnv, key_images: Vec<KeyImage>) -> ResponseResult {

cuprated: RPC handlers #355

Are you sure you want to change the base?

cuprated: RPC handlers #355

Conversation

hinto-janai commented Dec 5, 2024 • edited Loading

What

How

hinto-janai left a comment

Choose a reason for hiding this comment

RPC code structure

RPC handlers

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

1/2

Choose a reason for hiding this comment

2/2

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hinto-janai commented Dec 5, 2024 •

edited

Loading