BIP Draft: Silent Payment Output Script Descriptors

[craigraw]

This builds on and provides an alternative to #2026, which references an earlier discussion. Instead of modifying BIP352, a new output descriptor format is proposed in the style of BIPs 381-387.

Similar to #2026 key expressions starting with spscan and spspend are specified, but contain only version and key material information. The wallet birthday and additional labels are specified as optional additional arguments in the output descriptor format.

BIP352 authors: @josibake @RubenSomsen

Mail list discussion: https://groups.google.com/g/bitcoindev/c/bP6ktUyCOJI

[jonatack]

Looks sound and high quality for an initial draft. I don't see the BIP 2 specified mail list discussion for this proposal? (Edit: added to the PR description.)

[craigraw]

Thanks for the reminder - I've sent the mailing list post and added the link in the BIP. Also, the spaces after commas in the descriptors have been removed to be consistent with BIP 380-387.

[pythcoiner]

Thanks for working on this!
Overall lgtm, the only comment I have is, maybe a way to specify a label range could be useful?

[craigraw]

Overall lgtm, the only comment I have is, maybe a way to specify a label range could be useful?

I'm not strongly opposed to this, but given the cost of scanning for each additional label I'm reluctant to add it - it may make a wallet synchronization trivially impractical in a non-obvious way simply by accidentally specifying a large range.

[pythcoiner]

Maybe with adding a note discouraging wallet to export a range?
I mainly see range usefull for exchanges that can have a wide range, but also for this special case I guess they also need to backup associate metada for each label, so the descriptor is not sufficient anyway...

[craigraw]

There has been a second request for label ranges, and as I've said I'm not strongly opposed to adding this.

I suggest the following syntax - a hyphen between two integers represents an inclusive range:

sp(KEY,BIRTHDAY,1-10,15,20-25)

I think this is intuitive, compact and allows mixing sparse labels with ranges. There is slightly more parsing logic required now, and ranges will need to be validated (start < end), but this seems to be a comprehensive solution.

[pythcoiner]

I suggest the following syntax - a hyphen between two integers represents an inclusive range:

It makes sense

[jvgelder]

Overall lgtm, the only comment I have is, maybe a way to specify a label range could be useful?

I'm not strongly opposed to this, but given the cost of scanning for each additional label I'm reluctant to add it - it may make a wallet synchronization trivially impractical in a non-obvious way simply by accidentally specifying a large range.

Scanning cost is no longer the argument right? Do we still want arbitrary labels or do we want them to be consecutive? This would make it easier to parse in my opinion.

See label set scanning vs bip scanning benchmarks in : bitcoin-core/secp256k1#1765 (comment)

[murchandamus]

cc: @josibake, @RubenSomsen as authors of Silent Payments, @achow101 as author of most Descriptor BIPs.

[craigraw]

Label ranges in the style sp(KEY,BIRTHDAY,1-10,15,20-25) have been added in 62bb41f.

Do we still want arbitrary labels or do we want them to be consecutive? This would make it easier to parse in my opinion.

I don't think the parsing is complex enough to justify limiting labels to be consecutive only. There is a scanning cost (for light clients at least) and restricting potential use cases doesn't seem like the right approach.

[Eunovo]

Scanning cost is no longer the argument right? Do we still want arbitrary labels or do we want them to be consecutive? This would make it easier to parse in my opinion.

I previously thought it would be better only to specify the max label in the descriptor to keep it simple, but
Specifying label ranges in the descriptor could be useful for creating wallets that track transactions from specific sources. For example, a user can have two wallets that track different label ranges. I think that's enough reason to allow ranges.

[jvgelder]

Label ranges in the style sp(KEY,BIRTHDAY,1-10,15,20-25) have been added in 62bb41f.

Do we still want arbitrary labels or do we want them to be consecutive? This would make it easier to parse in my opinion.

I don't think the parsing is complex enough to justify limiting labels to be consecutive only. There is a scanning cost (for light clients at least) and restricting potential use cases doesn't seem like the right approach.

Ye make sense if we can get enough to agree I guess we need to revert some changes for #2026 too and add label ranges to stay compatible (unless we want to force descriptors ofc).

[murchandamus]

This looks pretty good from an editorial perspective. Perhaps some of the decisions around birthdate and labels could be collected into a Rationale section which currently doesn’t appear to be present. It would be good if more descriptor experts and silent payments experts could take a look at this, as I’m a bit of out of my depth regarding the nitty gritty details on those.

[achow101]

Sorry for the late review. I haven't really been following the current developments of silent payments, so it is possible that some of these thoughts are out of date.

The originally discussed sppub and spprv were not to be used in a descriptor but rather as a different idea. I don't think it is necessary to define a key silent payments key expression here.

The birthday is metadata and should not be included in a descriptor.

I'm unconvinced that labels belong in the descriptor too. Ultimately, that's very similar to ranged derivation paths which are explicitly unspecified in descriptors. It's up to the wallet loading the descriptor to choose the range.

[craigraw]

I don't think it is necessary to define a key silent payments key expression here.

Indeed, it is not strictly necessary. But I believe as wallet developers we should consider doing more than the minimum necessary where there justification to do so. Output descriptors are very much part of the usable interface of a modern wallet when they are displayed, shared and recorded, and there are significant benefits to a new key expression.

Consider the alternative - BIP380 gives us three key expressions: xpub/xprv, a hex-encoded public key, and a WIF. Since we don't want or need the chaincode or other HD derivation info, the xpub/xprv option is not desirable. That leaves sp(WIF,pubkey) and sp(WIF,WIF) - for example, sp(L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1, 0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600). There are several reasons why spscan/spspend is a meaningful improvement on this:

1. Easy identification of the type and sensitivity of the key expression. Differentiating between a WIF and hex-encoded public key is not simple for non-developers.
2. The advantages of Bech32m encoding, including error detection and unambiguous characters. This is particularly important for written backups, or those recorded on durable media.
3. The use of a single element that is simpler to understand with reduced user error from accidental or misguided mixing of keys.
4. A similar structure and appearance to xpubs, which are a commonly understood concept and user interface element making things simpler for users and developers alike.
5. WIF encoding/decoding is an additional requirement to support, as Bech32m is already required by BIP352.

If silent payments wallets are to be a major new type of wallet (alongside BIP32-style HD wallets), I believe the benefits listed above warrant a new key expression for a better user experience.

The birthday is metadata and should not be included in a descriptor.

This is an odd objection, since output descriptors are already metadata - data describing data. So a wallet birthday fits naturally into this definition. A better objection would be that previous output descriptor definitions have not included it. However, there are two significant differences to be considered:

1. The significant computational burden of scanning. It is reasonable to say that depending on the approach used and resources available, descriptors without a birthday may make wallet recovery practically impossible - particularly as the chain grows. Other descriptors do not share this very real user experience challenge.
2. Unlike other descriptors, silent payment descriptors require reference to the blockchain for the input pubkeys to create the outputs. Given this, an additional reference to the blockchain with the birthday block height is not unnatural.

I'm unconvinced that labels belong in the descriptor too. Ultimately, that's very similar to ranged derivation paths which are explicitly unspecified in descriptors. It's up to the wallet loading the descriptor to choose the range.

BIP352, unlike BIP44, does not define a gap limit when it comes to labels. This is probably intentional - recreating the problems associated with the gap limit is not desirable. Whatever the reason, given a documented method to "choose the range" does not exist, I believe the responsible approach must be to explicitly define which labels are in use.

[achow101]

I also don't see why spspend and spscan need to be the spend and scan keys concatenated for the encoding. This feels like something that was specified outside of descriptors beings shoehorned into descriptors (that's why WIF and extended keys are like that) when they can really just be 2 arguments to a single sp().

BIP 352 also defines derivation paths and not allowing other KEY expressions would preclude the use of those derivation paths.

Easy identification of the type and sensitivity of the key expression. Differentiating between a WIF and hex-encoded public key is not simple for non-developers.

In other discussions, possibly not transcribed, I have suggested that the scan key be instead called scan entropy and that the descriptor should store it as hex rather than a KEY expression. However, this does still somewhat conflict with the derivation path recommendation in BIP 352.

The advantages of Bech32m encoding, including error detection and unambiguous characters. This is particularly important for written backups, or those recorded on durable media.

IIRC this is actually one of the reasons that bech32 was never proposed for private keys since the error detection and correction has the possibility of false negatives and/or correcting to the wrong key if there are enough errors.

The use of a single element that is simpler to understand with reduced user error from accidental or misguided mixing of keys.

To me, this moves away from the ethos of descriptors. The purpose is to have descriptors that are fairly human readable. Concatenating multiple fields together to be encoded as a single element is antithetical to that.

A similar structure and appearance to xpubs, which are a commonly understood concept and user interface element making things simpler for users and developers alike.

In an ideal world, descriptors would not have supported xpubs. Rather there would have been a key expression like bip32(key, chaincode) rather than this single element mashing multiple things together. It is only because of the prevalence of extended keys that they needed to be supported.

WIF encoding/decoding is an additional requirement to support, as Bech32m is already required by BIP352.

WIF is already required in order to be compliant with BIP 380.

This is an odd objection, since output descriptors are already metadata - data describing data.

The philosophy behind descriptors is providing the data required to compute and spend an output script. A birthday is purely metadata in that regard - it does not provide any necessary information to generate the output scripts or the private keys required to spend them. That necessary information lives in the blockchain.

I believe the responsible approach must be to explicitly define which labels are in use.

This leads to a property where the silent payments descriptor is variable, and I find this property to also be antithetical to the purpose of descriptors.

If "choose a range" must be documented, that it can be done in a revision to BIP 352 or as a new BIP. I don't think that putting it in the descriptor itself is the correct place.

[craigraw]

I also don't see why spspend and spscan need to be the spend and scan keys concatenated for the encoding.

I think that fundamentally we disagree because we are coming from different points of view. I see output descriptors as part of the user interface of the wallet and thus should consider the practical benefits to the non-technical user. Reading your feedback, I see a more developer-centric view. While this view may reflect the original design goals of output descriptors, I question whether it is reflective of the actual usage of output descriptors in the real world today.

BIP 352 also defines derivation paths and not allowing other KEY expressions would preclude the use of those derivation paths.

I would rather say it precludes using unrelated derivation paths outside the BIP352 recommendations. The examples provided in this proposal do include derivation paths.

In other discussions, possibly not transcribed, I have suggested that the scan key be instead called scan entropy and that the descriptor should store it as hex rather than a KEY expression.

Perhaps I misunderstand what you are proposing, but it does not seem to make identification of the type and sensitivity of the key expression any clearer to non-developers (re the comment you were responding to).

IIRC this is actually one of the reasons that bech32 was never proposed for private keys since the error detection and correction has the possibility of false negatives and/or correcting to the wrong key if there are enough errors.

This seems a weak reason to avoid Bech32m. IMO automatic error correction should be avoided in production anyway. Hex-encoded public keys have no error detection at all and are strictly worse.

The purpose is to have descriptors that are fairly human readable. Concatenating multiple fields together to be encoded as a single element is antithetical to that.

To a developer familiar with BIP352, perhaps you are right. But I am fairly certain the majority of people actually using these descriptors will find that the type and sensitivity of the expressions

sp(spscan1qg3pk34hd8hkz2cq9kp30zqv6lz9nm06u2dazmrmq4rgfp84kemwsyxh6vlx0970ker49fqxx492gujl3y65uxp6dwd3lvngcgu9t5fljfww3ah)
sp(spspend1qg3pk34hd8hkz2cq9kp30zqv6lz9nm06u2dazmrmq4rgfp84kemwspy0n42kc0memf6zahf94p9xzc73au4c9v5g2dufq6u3xfr0fyqc82kukk)

are more human readable in type and sensitivity than

sp(L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1, 0260b2003c386519fc9eadf2b5cf124dd8eea4c4e68d5e154050a9346ea98ce600)
sp(L4rK1yDtCWekvXuE6oXD9jCYfFNV2cWRpVuPLBcCU2z8TrisoyY1, L5EZftvrYaSudiozVRzTqLcHLNDoVn7H5HSfM9BAN6tMJX8oTWz6)

In an ideal world, descriptors would not have supported xpubs. Rather there would have been a key expression like bip32(key, chaincode) rather than this single element mashing multiple things together. It is only because of the prevalence of extended keys that they needed to be supported.

And yet the single element has proven to be one of the most well known and usable elements in bitcoin wallets, where the type and sensitivity of the key expression is immediately identifiable. I think a world where bip32(key, chaincode) was used to represent both xpubs and xprvs would be have been in general less usable, and practically less secure.

Perhaps the best way to resolve this difference of opinion is to include both approaches of specifying the keys for a silent payments descriptor? This has the benefit of allowing full derivation path flexibility if desired, while maintaining a user-centric recommended approach for most cases.

WIF is already required in order to be compliant with BIP 380.

Yes, but full BIP 380 compliance is typically not a goal - wallets just support the parts that reflect the wallets they can actually create.

The philosophy behind descriptors is providing the data required to compute and spend an output script. A birthday is purely metadata in that regard - it does not provide any necessary information to generate the output scripts or the private keys required to spend them. That necessary information lives in the blockchain.

I would simply argue, again, that if it is practically impossible to recover a wallet without it, the birthday is necessary data. Ignoring this seems to favour one implementation approach over another. The checksum on an output descriptor also "does not provide any necessary information to generate the output scripts or the private keys required to spend them" yet obviously it is a reasonable and practical addition that recognises real world conditions are not ideal, and that accommodation must be made for this.

This leads to a property where the silent payments descriptor is variable, and I find this property to also be antithetical to the purpose of descriptors. If "choose a range" must be documented, that it can be done in a revision to BIP 352 or as a new BIP. I don't think that putting it in the descriptor itself is the correct place.

I actually misspoke - there is some reference to "choose a range" in Backup and Recovery. It is however completely impractical for light clients to scan 100k labels, so I don't find it of much value. Personally I dislike labels beyond the very specific exchange use case, and I think BIP32 accounts are a much better approach. I am only including labels here because others seem to see value, so if the consensus is to leave them out I will be happy to do so. I would encourage others to weigh in on this if they feel strongly either way.

[achow101]

Perhaps I misunderstand what you are proposing, but it does not seem to make identification of the type and sensitivity of the key expression any clearer to non-developers (re the comment you were responding to).

It removes the confusion that may occur when including something that looks like a private key in the public descriptor.

To a developer familiar with BIP352, perhaps you are right. But I am fairly certain the majority of people actually using these descriptors will find that the type and sensitivity of the expressions

sp(spscan1qg3pk34hd8hkz2cq9kp30zqv6lz9nm06u2dazmrmq4rgfp84kemwsyxh6vlx0970ker49fqxx492gujl3y65uxp6dwd3lvngcgu9t5fljfww3ah)
sp(spspend1qg3pk34hd8hkz2cq9kp30zqv6lz9nm06u2dazmrmq4rgfp84kemwspy0n42kc0memf6zahf94p9xzc73au4c9v5g2dufq6u3xfr0fyqc82kukk)

The main thing that is missing here is composability. sp() must be a top level descriptor, and that's fine. But as described in this BIP, none of the other KEY expressions can be used in it. If I already have a wallet where I wanted to use keys from that wallet in a new sp() descriptor, I would not be able to without having some tool to convert those keys into both spscan and spspend. Whereas allowing other KEY expressions would make this trivial. Furthermore, as new ways to produce keys emerge (e.g. musig2, frost), and those get defined into separate KEY expressions, it would be immensely useful to be able to compose those.

For example, musig() is a KEY expression. While I don't know off the top of my head if it is possible to use musig as the silent payments spend key, but being able to write sp(<scan key>, musig()) would be a very nice property. This is also more relevant for future KEY expressions that could come about with the express goal of being compatible with silent payments.

Fundamentally, I think the disconnect is that we are viewing descriptors as being useful for different things. You value it as a tool for creating backups, while I value it as a tool for constructing interesting scripts. This proposal satisfies the former, while neglecting the latter.

Also, I believe the proposed spscan and spspend exceed the 89 character limit specified in BIP 173 so it does not have all of the same error detection properties. Additionally, the descriptor checksum itself already has error detection properties, and IIRC is better suited for long strings.

I would simply argue, again, that if it is practically impossible to recover a wallet without it, the birthday is necessary data.

Sure, but this is Output Script Descriptors, not Wallet Descriptors. I agree that the birthday is useful information to a wallet when recovering, but descriptors is not, and never has been, enough information to fully recover a wallet.

Birthdays are also useful for descriptors outside of silent payments. I think it's more useful to have a "wallet backups with descriptors" bip which describes how birthdays can be included alongside a descriptor. Doing this generically seems like it would be much more useful.

The checksum on an output descriptor also "does not provide any necessary information to generate the output scripts or the private keys required to spend them" yet obviously it is a reasonable and practical addition that recognises real world conditions are not ideal, and that accommodation must be made for this.

I don't think that argument makes any sense. Checksums are a necessary part of encodings so this isn't so much a property of defining output scripts but rather an encoding. Also the checksum was not part of the design for descriptor originally and only bolted on afterwards.

[pythcoiner]

that if it is practically impossible to recover a wallet without it,

while I'm in favor having birthday/range included in the descriptor, I don't consider it to be impossible to recover the wallet without it, it's just a performance/UX improvement from my pov.

[craigraw]

You value it as a tool for creating backups, while I value it as a tool for constructing interesting scripts. This proposal satisfies the former, while neglecting the latter.

I understand your point of view better now. I notice you did not respond to my suggestion of supporting both approaches to specifying the scan and spend keys?

Birthdays are also useful for descriptors outside of silent payments. I think it's more useful to have a "wallet backups with descriptors" bip which describes how birthdays can be included alongside a descriptor. Doing this generically seems like it would be much more useful.

Requiring an industry to adopt an entirely new format to add a single integer seems to me somewhat dogmatic. But I'm not going to die on this hill - if there is no support for adding a birthday I will remove it.

[craigraw]

I don't consider it to be impossible to recover the wallet without it, it's just a performance/UX improvement from my pov.

Removing the word "practically" from my statement changes its meaning substantially. Try catching up a few months on any of the current silent payments mobile wallets, and then imagine what it would be like to catch up several years.

[pythcoiner]

Removing the word "practically" from my statement changes its meaning substantially.

My bad. I mean I don't consider it practically impossible neither in a normal "common" case tbh.

Try catching up a few months on any of the current silent payments mobile wallets, and then imagine what it would be like to catch up several years.

By "common" case I'll take example of data you shared on this delving post:

Here is are some benchmarking figures using frigate on an 8x RTX 3060 GPU machine testing over a 64 week period:

	Time (ms)	Tx/sec
No label	16,949	4,199,074
Change label	17,042	4,176,159
10 labels	19,334	3,681,086
30 labels	25,109	2,834,446

napkin math: lets take 500k txs daily with a wallet that uses 30 labels, it gives us what, 1+ minutes per year? (365 * 500_000 / 2_834_446 ~= 64 secs)

did I overlook something?

[craigraw]

did I overlook something?

Frigate uses a server-based approach to scanning, with GPU optimisation if the hardware is available. If you use the light client method described in BIP352, the client needs to download, parse and compute the data, which turns out to be many orders of magnitude slower, especially on a mobile phone.

[achow101]

I notice you did not respond to my suggestion of supporting both approaches to specifying the scan and spend keys?

Supporting both seems like it would be more confusing.

Requiring an industry to adopt an entirely new format to add a single integer seems to me somewhat dogmatic.

I don't mean a BIP just for birthdays, but a more generic BIP for backups that can include birthdays and other metadata that is useful. For me, I think that would also be an appropriate place to put derivation and label ranges.

[pythcoiner]

but a more generic BIP for backups that can include birthdays and other metadata that is useful.

I'm working on a draft BIP based on the backup format we implemented in Liana, but I still see some advantage to having minimal metadata (in this case, the birthday) included in the descriptor itself.

A complete wallet backup needs frequent updates as things are added or change (labels, key aliases, etc.), but these aren't necessary to find your coins. The birthday, on the other hand, can't change and is helpful for finding coins.

The derivation range is somewhere in between—it's helpful but does change. Still, from a wallet dev perspective, I see real value in including it in the descriptor: any wallet supporting the descriptor format will have access to it, whereas not all wallets will implement support for a separate backup format.