feat: integration of frigate ephemeral scanning by sdmg15 · Pull Request #48 · bitcoindevkit/bdk-sp

sdmg15 · 2025-12-22T20:56:38Z

Description

This PR integrates the experimental Frigate scanning server with ephemeral client keys.

Notes to the reviewers

The specification and how to run a frigate server can be found at: https://github.com/sparrowwallet/frigate

Changelog notice

Checklists

All Submissions:

I've signed all my commits
I followed the conventional commit guidelines
I ran just p (fmt, clippy and test) before committing

nymius

I have tried the command without success. Could you provide me some examples to try?

If you could create an executable step by step example like the built in doc/tabconf7 would be great.

Notice it builds a container with frigate on it. You only have to expose the ports to be able to communicate with it from outside of the container.

cli/v2/src/main.rs

sdmg15 · 2026-01-23T18:35:12Z

Thanks for a quick round of review, I noticed I have some unpushed changes which I just did.
I tried the containers commands but they took too long to build and I thought I'd just test "manually" by running frigate and rpc outside of it. But I'll give it another try from there will give you the commands to run.

sdmg15 · 2026-02-06T09:35:24Z

@nymius the latest commit 6518059 is in a testable state.
I wasn't able to use the regtest playbook, due to issues mentionned in #49.

The only new command to what is inside regtest_playbook.sh is:
cargo run scan-frigate --url 127.0.0.1:57001 --start 0 assuming that that the Frigate client is running on port 57001

nymius

I fixed the issue with bdk-tx. You should rebase.

Successfully executed scanning on regtest:

$ REGTEST_ADDRESS=$(regtest-bdk unused_address | jq -r '.address' | tr -d '\n')
$ just mine 1 $REGTEST_ADDRESS
[
  "5e724d83a5e8999874ec679ed5823962f56c82bf4810be937b248dc5a08b15cb"
]
$ just mine 101 2>&1 >/dev/null
$ regtest-bdk sync
{}
$ regtest-bdk balance
{
  "satoshi": {
    "confirmed": 5000000000,
    "immature": 0,
    "trusted_pending": 0,
    "untrusted_pending": 0
  }
}
$ SP_CODE=$(regtest-sp code | jq -r '.silent_payment_code' | tr -d '\n')
$ RAW_TX=$(regtest-bdk create_sp_tx --to-sp $SP_CODE:10000 --fee_rate 5 | jq -r '.raw_tx' | tr -d '\n')
$ TXID=$(regtest-bdk broadcast --tx $RAW_TX | jq -r '.txid' | tr -d '\n')
$ just mine 1
[
  "2c28f3fa2cdd742a3e5e564f770802c558324506ea6db3ab6c0cea85ab83c0c3"
]
$ regtest-bdk sync
{}
$ regtest-bdk balance
{
  "satoshi": {
    "confirmed": 4999989228,
    "immature": 0,
    "trusted_pending": 0,
    "untrusted_pending": 0
  }
}
$ regtest-sp scan-frigate --url 127.0.0.1:57001 --start 0
2026-02-12T19:10:12.162809Z  WARN bdk_sp_oracles::frigate: Read bytes from stream 59
2026-02-12T19:10:12.183998Z  WARN bdk_sp_oracles::frigate: Read bytes from stream 156
2026-02-12T19:10:12.184062Z  INFO bdk_sp_oracles::frigate: Subscribed to silent payment address: String("sprt1qqvf42gn2770nwf8jvalczzv80ad8p9zseahnleklzzklqxkg5s80wqa7xxx58l2wvumjlmrzy9eed4vsd647jcmp6jg5tr08unyhex7u5chnkk64")
2026-02-12T19:10:12.184092Z  INFO sp_cli2: Starting frigate scanning loop...
2026-02-12T19:10:12.361669Z  WARN bdk_sp_oracles::frigate: Read bytes from stream 461
2026-02-12T19:10:12.414640Z  WARN bdk_sp_oracles::frigate: Read bytes from stream 200
2026-02-12T19:10:12.474964Z  WARN bdk_sp_oracles::frigate: Read bytes from stream 448
2026-02-12T19:10:12.475701Z  INFO sp_cli2: Progress 1
2026-02-12T19:10:12.475740Z  WARN sp_cli2: Scanning completed
$ echo $SP_CODE
sprt1qqvf42gn2770nwf8jvalczzv80ad8p9zseahnleklzzklqxkg5s80wqa7xxx58l2wvumjlmrzy9eed4vsd647jcmp6jg5tr08unyhex7u5chnkk64
$ regtest-sp balance
{
  "confirmed": {
    "immature": 0,
    "spendable": 0,
    "total": 0
  }
}
{
  "unconfirmed": {
    "total": 10000,
    "trusted": 0,
    "untrusted": 10000
  }
}

Notice the balance appears as unconfirmed because the block to which the transaction is anchored doesn't belong to the wallet local chain. I left a comment in the code mentioning how to fix that.

I think that Read bytes from stream {bytes} belongs to the DEBUG scope, rather than WARN. I would also print a one line log of the balance, after each execution of the command.

Could you give me more information about the issues that are preventing you from testing with the playbooks? Could you setup nix?

Good work!

nymius · 2026-02-12T19:24:04Z

cli/v2/src/main.rs

+                    for secret in secrets_by_height.into_iter().filter(|v| v.0 > 0) {
+                        // Since frigate doesn't provide a blockchain.getblock we will mimick that here
+                        // By constructing a block from the block header and the list of transactions
+                        // received from the scan request
+                        let mut raw_blk = client.get_block_header(secret.0).await.unwrap();
+                        raw_blk.push_str("00");
+
+                        // Push dummy coinbase
+                        let dummy_coinbase = "01000000010000000000000000000000000000000000000000000000000000000000000000ffffffff1b03951a0604f15ccf5609013803062b9b5a0100072f425443432f20000000000000000000";
+                        let coinbase: Transaction =
+                            deserialize(&Vec::<u8>::from_hex(dummy_coinbase).unwrap()).unwrap();
+                        let mut block: Block =
+                            deserialize(&Vec::<u8>::from_hex(&raw_blk).unwrap()).unwrap();
+
+                        let mut txs: Vec<Transaction> = vec![coinbase];
+                        for key in secret.1.keys() {
+                            let raw_tx = client.get_transaction(key.to_string()).await.unwrap();
+                            let tx: Transaction =
+                                deserialize(&Vec::<u8>::from_hex(&raw_tx).unwrap()).unwrap();
+                            txs.push(tx);
+                        }
+
+                        block.txdata = txs;
+                        tracing::debug!("Final block {:?}", block);
+                        wallet.apply_block_relevant(&block, secret.1, secret.0);
+                    }


Maybe we should improve the wallet interface to avoid this, but it works.
You're missing the chain update to make the wallet aware of the block this transaction belongs to. If not, the balance of transactions is going to be unconfirmed for ever.

let checkpoint = wallet.chain().tip().insert(BlockId { height, hash }); wallet.update_chain(checkpoint);

nymius · 2026-02-12T19:26:31Z

oracles/src/frigate/mod.rs

+}
+
+const SUBSCRIBE_RPC_METHOD: &str = "blockchain.silentpayments.subscribe";
+const _UNSUBSCRIBE_RPC_METHOD: &str = "blockchain.silentpayments.unsubscribe";


We don't have long enough wait times in regtest to suffer from request taking minutes, but I would add a sane default to cut execution if no progress has been made in enough time.

nymius · 2026-02-12T19:30:11Z

cli/v2/src/main.rs

+                            .or_insert(HashMap::from([(h.tx_hash, h.tweak_key)]));
+                    });
+
+                    // Filter when the height is 0, because that would mean mempool transaction


I would add a comment mentioning this paragraph from Frigate's README, and how we are not following that approach here, and what this alternative is about.

I'm not sure I fully get the comment. But it's referenced here https://github.com/sparrowwallet/frigate/tree/master?tab=readme-ov-file#notifications

height: The integer height of the block the transaction was confirmed in. For mempool transactions, 0 should be used.

Sorry, I forgot to paste the paragraph:

Clients should retrieve the transactions listed in the history with blockchain.transaction.get and subscribe to all owned outputs with blockchain.scripthash.subscribe. Electrum wallet functionality then proceeds as normal. In other words, the silent payments address subscription is a replacement for the monotonically increasing derivation path index in BIP32 wallets. The subscription seeks only to add to the client's knowledge of incoming silent payments transactions. The client is responsible for checking the transactions do actually send to addresses it has keys for, and using normal Electrum wallet synchronization techniques to monitor for changes to these addresses. The tweak key is provided to allow the client to avoid looking up the scriptPubKeys of spent outputs.

Yes we're already getting the transactions details with blockchain.transaction.get the only thing we're not doing here is blockchain.scripthash.subscribe because cmiiw we're just doing one time scanning so I didn't find it necessary to subscribe to the scripthash.

I know, but I say it will be better to document the difference with the suggested approach.

Also, this gave me an idea for a greater architectural change that may be worth exploring. To keep a queue of the blocks matched, and any other data given by the sp oracle (like txhash), and synchronize on demand by requesting this blocks to different sources. This would create a two stage synchronization process for light clients:

Find matching blocks: frigate, blindbit + cbf, rpc

Request matching blocks and finalize scanning: electrum, cbf, rpc, p2p

Yeah interesting. We could discuss further on that once this one is merged and I can give it a try it that's fine by you.

Btw will give you the full reports on what's preventing me from using the playbook and address the comments in the next days. Thanks.

sdmg15 force-pushed the feat/frigate branch from c72dc84 to ceb8420 Compare January 12, 2026 20:21

nymius reviewed Jan 23, 2026

View reviewed changes

cli/v2/src/main.rs Outdated Show resolved Hide resolved

cli/v2/src/main.rs Outdated Show resolved Hide resolved

cli/v2/src/main.rs Outdated Show resolved Hide resolved

nymius assigned sdmg15 Jan 23, 2026

nymius added the enhancement New feature or request label Jan 23, 2026

integration of frigate ephemeral scanning

732ca71

sdmg15 force-pushed the feat/frigate branch from ceb8420 to 732ca71 Compare January 23, 2026 18:31

sdmg15 marked this pull request as ready for review January 23, 2026 18:32

sdmg15 requested a review from nymius February 6, 2026 09:28

feat: improve listening to tcp stream

6518059

sdmg15 force-pushed the feat/frigate branch from c4132e8 to 6518059 Compare February 6, 2026 09:30

nymius reviewed Feb 12, 2026

View reviewed changes

Conversation

sdmg15 commented Dec 22, 2025

Description

Notes to the reviewers

Changelog notice

Checklists

All Submissions:

Uh oh!

nymius left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sdmg15 commented Jan 23, 2026

Uh oh!

sdmg15 commented Feb 6, 2026

Uh oh!

nymius left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants