Support paying static invoices #3140

valentinewallace · 2024-06-20T20:49:08Z

Support sending async payments as an always-online sender per lightning/bolts#1149, minus including the invreq in the payment onion.

Async receive is not yet supported, nor is sending async payments as an often-offline sender. Upcoming PRs will add async receive which will allow us to test the flow end-to-end.

Blocked on ~~#3125~~, ~~#3145~~, ~~#3085~~.

lightning/src/ln/channelmanager.rs

codecov-commenter · 2024-06-24T16:22:27Z

Codecov Report

Attention: Patch coverage is 85.78199% with 30 lines in your changes missing coverage. Please review.

Project coverage is 90.02%. Comparing base (1059f5f) to head (6e27aec).
Report is 32 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/ln/channelmanager.rs	50.00%	16 Missing and 1 partial ⚠️
lightning/src/ln/outbound_payment.rs	94.93%	8 Missing ⚠️
lightning/src/onion_message/messenger.rs	0.00%	2 Missing ⚠️
lightning/src/ln/peer_handler.rs	0.00%	1 Missing ⚠️
lightning/src/offers/signer.rs	93.33%	0 Missing and 1 partial ⚠️
lightning/src/onion_message/functional_tests.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3140      +/-   ##
==========================================
+ Coverage   89.63%   90.02%   +0.38%     
==========================================
  Files         126      126              
  Lines      102383   104805    +2422     
  Branches   102383   104805    +2422     
==========================================
+ Hits        91776    94350    +2574     
+ Misses       7877     7750     -127     
+ Partials     2730     2705      -25

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

valentinewallace · 2024-07-08T17:44:25Z

Rebased so I could get #3156 in the follow-up.

TheBlueMatt · 2024-07-10T13:22:45Z

Needs rebase.

lightning/src/offers/static_invoice.rs

lightning/src/ln/channelmanager.rs

lightning/src/ln/outbound_payment.rs

lightning/src/ln/channelmanager.rs

lightning/src/ln/outbound_payment.rs

TheBlueMatt · 2024-07-10T23:55:27Z

lightning/src/onion_message/messenger.rs

@@ -1463,7 +1463,7 @@ where
 						let context = match context {
 							None => OffersContext::Unknown {},
 							Some(MessageContext::Offers(context)) => context,
-							Some(MessageContext::Custom(_)) => {
+							Some(MessageContext::Custom(_)) | Some(MessageContext::AsyncPayments(_)) => {


Why are offers messages allowed to have an async context? Some docs on AsyncPaymentsContext would be nice that say where they're allowed and what they're used for.

Added docs.

Why are offers messages allowed to have an async context?

They shouldn't be, hence the debug assert below. Lmk if I'm missing something.

To rid ourselves of the assert, we need a larger refactor such that ParsedOnionMessageContents (or something like it) contains the appropriate context in each variant. Out of scope of this PR, though.

We can probably use a _ catchall instead, though. Same elsewhere.

TheBlueMatt · 2024-07-10T23:57:24Z

lightning/src/onion_message/messenger.rs

@@ -1473,14 +1473,22 @@ where
 					},
 					#[cfg(async_payments)]
 					ParsedOnionMessageContents::AsyncPayments(AsyncPaymentsMessage::HeldHtlcAvailable(msg)) => {
+						debug_assert!(context.is_none());


While we do encrypt the context, I don't believe we MAC it, potentially allowing someone creating a blinded path to us to brute-force their way into this branch (they only have to get a few bytes to match). Same below for the debug assertion.

The other branches have debug asserts because the correctness is checked in peel_onion_message (and their assert messages should really be updated to mention that!), but you don't do that here.

Fixed peeled_onion_message and removed this assertion. One thing I'm not sure about is whether to fail held_htlc_available messages that have Some context, since we expect None. I went with ignoring the context for now since we would've failed to decode it if it contained unknown even TLVs.

lightning/src/ln/outbound_payment.rs

lightning/src/blinded_path/message.rs

lightning/src/ln/channelmanager.rs

TheBlueMatt · 2024-07-11T00:23:31Z

lightning/src/ln/channelmanager.rs

+			invoice, payment_id, &*self.entropy_source
+		)?;
+
+		let mut pending_async_payments_messages = self.pending_async_payments_messages.lock().unwrap();


Should we consider an Event at this point to tell the user the payment is pending+async and not to expect resolution immediately?

Makes sense, tracking this todo on #2298.

lightning/src/ln/outbound_payment.rs

lightning/src/offers/static_invoice.rs

valentinewallace · 2024-09-13T14:38:39Z

Squashed in the new changes with the following diff:

diff --git a/lightning/src/offers/signer.rs b/lightning/src/offers/signer.rs
index d6372bca8..d8caa2175 100644
--- a/lightning/src/offers/signer.rs
+++ b/lightning/src/offers/signer.rs
@@ -44,7 +44,7 @@ const OFFER_PAYMENT_ID_HMAC_INPUT: &[u8; 16] = &[5; 16];
 const ASYNC_PAYMENT_ID_HMAC_INPUT: &[u8; 16] = &[6; 16];

 // HMAC input for a `PaymentHash`. The HMAC is used in `OffersContext::InboundPayment`.
-const PAYMENT_HASH_HMAC_INPUT: &[u8; 16] = &[6; 16];
+const PAYMENT_HASH_HMAC_INPUT: &[u8; 16] = &[7; 16];

 /// Message metadata which possibly is derived from [`MetadataMaterial`] such that it can be
 /// verified.
diff --git a/lightning/src/offers/static_invoice.rs b/lightning/src/offers/static_invoice.rs
index 74955abe0..d083cae88 100644
--- a/lightning/src/offers/static_invoice.rs
+++ b/lightning/src/offers/static_invoice.rs
@@ -315,8 +315,10 @@ impl StaticInvoice {
        }

        pub(crate) fn from_same_offer(&self, invreq: &InvoiceRequest) -> bool {
-               let invoice_offer_tlv_stream = TlvStream::new(&self.bytes).range(OFFER_TYPES);
-               let invreq_offer_tlv_stream = TlvStream::new(invreq.bytes()).range(OFFER_TYPES);
+               let invoice_offer_tlv_stream = TlvStream::new(&self.bytes).range(OFFER_TYPES)
+                       .map(|tlv_record| tlv_record.record_bytes);
+               let invreq_offer_tlv_stream = TlvStream::new(invreq.bytes()).range(OFFER_TYPES)
+                       .map(|tlv_record| tlv_record.record_bytes);
                invoice_offer_tlv_stream.eq(invreq_offer_tlv_stream)
        }
 }

Useful for ensuring that an inbound static invoice matches one of our outbound invreqs, otherwise it is an unexpected invoice and should be ignored and not paid.

Adds a pending outbound payment variant for async payments, which indicates that we have received a static invoice to pay and have generated a keysend preimage for the eventual payment. When the recipient comes back online, we'll transition from this new state to Retryable and actually forward the HTLCs.

Allows us to repurpose this util to create blinded paths in an async payments message context while maintaining support for the offers context.

Supported when the sender is an always-online node. Here we send the initial held_htlc_available onion message upon receipt of a static invoice, next we'll need to actually send HTLCs upon getting a response to said OM.

Because we may receive a static invoice to pay days before the recipient actually comes back online to receive the payment, it's good to do as many checks as we can up-front. Here we ensure that the blinded paths provided in the invoice won't cause us to exceed the maximum onion packet size.

This new util will be able to send to both static and non-static BOLT 12 invoices.

Async receive is not yet supported. Here we process inbound release_htlc onion messages, check that they actually correspond to one of our outbound payments, and actually forward the HTLCs. Valid release_htlc receipt indicates that the recipient has now come online to receive.

Async payments may have very high expires because we may be waiting for days for the recipient to come online, so it's important that users be able to abandon these payments early if needed.

If someone sends us an unexpected or duplicate release_held_htlc onion message, we should simply ignore it and not persist the entire ChannelManager in response.

We want to specify that these methods are only to be used in an outbound offers payment context, because we'll be adding similar methods for the outbound async payments context in upcoming commits.

See AsyncPaymentsContext::hmac, but this prevents the recipient from deanonymizing us. Without this, if they are able to guess the correct payment id, then they could create a blinded path to us and confirm our identity. We also move the PAYMENT_HASH_HMAC_INPUT const to use &[7; 16], which is safe because this const was added since the last release. This ordering reads more smoothly.

Currently used when initiating an async payment via held_htlc_available OM. This OM needs a reply path back to us, so use this error for our invoice_error OM if we fail to create said reply path.

While these variants may sound similar, they are very different. One is so temporary it's never even persisted to disk, the other is a state we will stay in for hours or days. See added docs for more info.

This field isn't necessary because we already authenticate the messages via the blinded reply paths payment_id, nonce and HMAC.

valentinewallace · 2024-09-13T14:40:34Z

Oops, and a rustfmt fix:

diff --git a/lightning/src/offers/static_invoice.rs b/lightning/src/offers/static_invoice.rs
index d083cae88..4910c57c5 100644
--- a/lightning/src/offers/static_invoice.rs
+++ b/lightning/src/offers/static_invoice.rs
@@ -315,9 +315,11 @@ impl StaticInvoice {
        }

        pub(crate) fn from_same_offer(&self, invreq: &InvoiceRequest) -> bool {
-               let invoice_offer_tlv_stream = TlvStream::new(&self.bytes).range(OFFER_TYPES)
+               let invoice_offer_tlv_stream = TlvStream::new(&self.bytes)
+                       .range(OFFER_TYPES)
                        .map(|tlv_record| tlv_record.record_bytes);
-               let invreq_offer_tlv_stream = TlvStream::new(invreq.bytes()).range(OFFER_TYPES)
+               let invreq_offer_tlv_stream = TlvStream::new(invreq.bytes())
+                       .range(OFFER_TYPES)
                        .map(|tlv_record| tlv_record.record_bytes);
                invoice_offer_tlv_stream.eq(invreq_offer_tlv_stream)
        }

jkczyz · 2024-09-13T15:59:55Z

lightning/src/offers/merkle.rs

@@ -249,6 +249,7 @@ impl<'a> TlvStream<'a> {
 }

 /// A slice into a [`TlvStream`] for a record.
+#[derive(Eq, PartialEq)]


nit: We can drop the derive now , too.

TheBlueMatt

A few things that will need following up on before we ship this, but good enough for landing behind a cfg flag.

TheBlueMatt · 2024-09-12T16:24:32Z

lightning/src/blinded_path/message.rs

+		///
+		/// [`ReleaseHeldHtlc`]: crate::onion_message::async_payments::ReleaseHeldHtlc
+		/// [`HeldHtlcAvailable`]: crate::onion_message::async_payments::HeldHtlcAvailable
+		nonce: Nonce,


Please don't have commits that rewrite things done earlier in the same PR.

TheBlueMatt · 2024-09-13T19:58:40Z

lightning/src/ln/outbound_payment.rs

 	InvoiceReceived {
 		payment_hash: PaymentHash,
 		retry_strategy: Retry,
 		// Note this field is currently just replicated from AwaitingInvoice but not actually
 		// used anywhere.
 		max_total_routing_fee_msat: Option<u64>,
 	},
+	// This state applies when we are paying an often-offline recipient and another node on the


Why are these comments not being added in the same commit that adds StaticInvoiceReceived? Also, presumably they should be doc comments (for those who look at internal docs, or use an IDE...though it doesn't matter to me).

TheBlueMatt · 2024-09-13T20:00:58Z

lightning/src/ln/outbound_payment.rs

+				} => {
+					let invreq = &retryable_invoice_request
+						.as_ref()
+						.ok_or(Bolt12PaymentError::UnexpectedInvoice)?


This is always gonna fail, right? RetryableInvoiceRequest is take()n pretty quickly after building the AwaitingInvoice to retry sending the invreq, so by the time we get an invoice its usually gone.

Ugh, sorry... I should have caught this. Do we want to hold on to the invoice request indefinitely? Maybe that would useful in the future for proof-of-payer? Alternatively, we could store the OfferId instead.

Probably have to, right? We want to include the invreq in the payment HTLC.

TheBlueMatt · 2024-09-13T20:13:50Z

lightning/src/ln/channelmanager.rs

@@ -11106,7 +11106,7 @@ where
 	fn release_held_htlc(&self, _message: ReleaseHeldHtlc, _context: AsyncPaymentsContext) {}

 	fn release_pending_messages(&self) -> Vec<(AsyncPaymentsMessage, MessageSendInstructions)> {
-		Vec::new()
+		core::mem::take(&mut self.pending_async_payments_messages.lock().unwrap())


Why isn't this in the commit that adds pending_async_payments_messages?

TheBlueMatt · 2024-09-15T13:58:59Z

lightning/src/ln/channelmanager.rs

@@ -10943,6 +10943,35 @@ where
 		let secp_ctx = &self.secp_ctx;
 		let expanded_key = &self.inbound_payment_key;

+		macro_rules! handle_pay_invoice_res {


Why is this a separate commit? Why not just squash the macro into the previous commit instead of having non-DRY code at all?

TheBlueMatt · 2024-09-15T16:17:52Z

lightning/src/ln/outbound_payment.rs

@@ -967,6 +968,10 @@ impl OutboundPayments {
 					if !invoice.from_same_offer(invreq) {
 						return Err(Bolt12PaymentError::UnexpectedInvoice)
 					}
+					if invoice.invoice_features().requires_unknown_bits_from(&features) {


Why is this in a separate commit? Its exclusively updating code that was added in Support initiating an async payment to a static invoice., same for Set max path len on receipt of static invoice.

TheBlueMatt · 2024-09-15T16:48:22Z

lightning/src/ln/outbound_payment.rs

@@ -1725,6 +1907,22 @@ impl OutboundPayments {
 					true
 				}
 			},
+			PendingOutboundPayment::StaticInvoiceReceived { route_params, payment_hash, .. } => {
+				let is_stale =
+					route_params.payment_params.expiry_time.unwrap_or(u64::MAX) <


This is entirely set based on the static invoice, which means we may hold a pending outbound HTLC for a year or two if that's what the static invoice's expiry time is. We need a way to actually time these out. Maybe as a part of #3262 (but we shouldn't let it be a footgun where we hold a pending payment forever if a user forgets to override the value/maybe we should have a default of 1 week).

TheBlueMatt · 2024-09-15T16:49:03Z

lightning/src/ln/outbound_payment.rs

+			PaymentParameters::from_bolt12_invoice(&invoice), invoice.amount_msats()
+		);
+		if let Some(max_fee_msat) = max_total_routing_fee_msat {
+			route_params.max_total_routing_fee_msat = Some(max_fee_msat);


We need a way to set a max total routing fee that is non-default on static invoice payments, though this isn't new, we need it generally for bolt 12 (#3262).

TheBlueMatt · 2024-09-15T16:56:23Z

lightning/src/onion_message/async_payments.rs

-pub struct ReleaseHeldHtlc {
-	/// Used to release the HTLC held upstream if it matches the corresponding
-	/// [`HeldHtlcAvailable::payment_release_secret`].
-	pub payment_release_secret: [u8; 32],


This commit could have gone first to avoid rewriting a bunch of code that was added in this PR, though I get that it was added late in the PR's life.

valentinewallace added the blocked on dependent pr label Jun 20, 2024

valentinewallace commented Jun 20, 2024

View reviewed changes

lightning/src/ln/channelmanager.rs Outdated Show resolved Hide resolved

valentinewallace commented Jun 24, 2024

View reviewed changes

lightning/src/ln/channelmanager.rs Show resolved Hide resolved

valentinewallace force-pushed the 2024-06-pay-static-invoice branch 2 times, most recently from f316229 to 6eb3711 Compare June 24, 2024 16:15

jkczyz self-requested a review June 24, 2024 19:54

valentinewallace force-pushed the 2024-06-pay-static-invoice branch from 6eb3711 to 175d7d5 Compare June 27, 2024 15:42

valentinewallace force-pushed the 2024-06-pay-static-invoice branch 2 times, most recently from d75a5d3 to 635a3e6 Compare July 8, 2024 17:43

valentinewallace force-pushed the 2024-06-pay-static-invoice branch 2 times, most recently from 4f8fe07 to 51302e8 Compare July 8, 2024 18:17

valentinewallace removed the blocked on dependent pr label Jul 9, 2024

valentinewallace force-pushed the 2024-06-pay-static-invoice branch 3 times, most recently from 2d0e7a7 to f0d9fde Compare July 10, 2024 19:17

jkczyz reviewed Jul 10, 2024

View reviewed changes

valentinewallace force-pushed the 2024-06-pay-static-invoice branch 2 times, most recently from 2ab6fe8 to bd29d90 Compare July 10, 2024 20:46

TheBlueMatt reviewed Jul 11, 2024

View reviewed changes

valentinewallace mentioned this pull request Jul 12, 2024

Async Payments #2298

Open

22 tasks

valentinewallace force-pushed the 2024-06-pay-static-invoice branch 2 times, most recently from df323f3 to 1d6ba59 Compare July 12, 2024 21:34

valentinewallace force-pushed the 2024-06-pay-static-invoice branch 2 times, most recently from fe21bc4 to 60c390f Compare July 26, 2024 18:49

valentinewallace mentioned this pull request Jul 26, 2024

Include invoice requests in async payment onions #3207

Merged

2 tasks

valentinewallace force-pushed the 2024-06-pay-static-invoice branch 2 times, most recently from 1ac6a17 to 5b8c21e Compare July 31, 2024 00:29

valentinewallace force-pushed the 2024-06-pay-static-invoice branch from 9d71600 to 8ab04dc Compare September 13, 2024 14:37

valentinewallace added 20 commits September 13, 2024 10:40

Support checking that a static invoice matches an outbound invreq.

cff6e34

Useful for ensuring that an inbound static invoice matches one of our outbound invreqs, otherwise it is an unexpected invoice and should be ignored and not paid.

Support creating PaymentParameters from static invoices.

ad63a70

Pass full message context into ChanMan blinded path util.

7fb16ea

Allows us to repurpose this util to create blinded paths in an async payments message context while maintaining support for the offers context.

Release pending async payments to PeerManager.

c976e4c

Support initiating an async payment to a static invoice.

b6f4479

Supported when the sender is an always-online node. Here we send the initial held_htlc_available onion message upon receipt of a static invoice, next we'll need to actually send HTLCs upon getting a response to said OM.

DRY handling when initiating payment to BOLT 12 invoice.

28269a7

Error on static invoice with unknown required features.

e4d7681

Split off send_payment_for_bolt12_invoice_internal util.

69356e7

This new util will be able to send to both static and non-static BOLT 12 invoices.

Timeout expired outbound async payments.

985e6ac

Support abandoning pending outbound async payments.

6d415b1

Async payments may have very high expires because we may be waiting for days for the recipient to come online, so it's important that users be able to abandon these payments early if needed.

Correct docs on payment id in RecentPaymentDetails.

7dd1787

Don't trigger manager persistence on unexpected release_htlc message.

c4f3e25

If someone sends us an unexpected or duplicate release_held_htlc onion message, we should simply ignore it and not persist the entire ChannelManager in response.

Rename Payment{Hash,Id} hmac creation/verification methods for offers.

5a7f523

We want to specify that these methods are only to be used in an outbound offers payment context, because we'll be adding similar methods for the outbound async payments context in upcoming commits.

Add new Bolt12PaymentError for failed blinded path creation.

26d1582

Currently used when initiating an async payment via held_htlc_available OM. This OM needs a reply path back to us, so use this error for our invoice_error OM if we fail to create said reply path.

Document PendingOutboundPayment::{Static}InvoiceReceived semantics.

4bcf53e

While these variants may sound similar, they are very different. One is so temporary it's never even persisted to disk, the other is a state we will stay in for hours or days. See added docs for more info.

Remove payment_release_secret from async payments messages.

6e27aec

This field isn't necessary because we already authenticate the messages via the blinded reply paths payment_id, nonce and HMAC.

valentinewallace force-pushed the 2024-06-pay-static-invoice branch from 8ab04dc to 6e27aec Compare September 13, 2024 14:40

jkczyz approved these changes Sep 13, 2024

View reviewed changes

TheBlueMatt approved these changes Sep 15, 2024

View reviewed changes

TheBlueMatt merged commit 22146a9 into lightningdevkit:main Sep 15, 2024
18 of 21 checks passed

This was referenced Oct 18, 2024

TABConf 6 Socratic Session: Lightning Protocol Development TriangleBitDevs/TriangleBitDevs.github.io#28

Closed

Socratic Session: Lightning BOLT Spec and Implementation Updates TABConf/6.tabconf.com#148

Closed

valentinewallace mentioned this pull request Nov 14, 2024

Add static invoice creation utils to ChannelManager #3408

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support paying static invoices #3140

Support paying static invoices #3140

valentinewallace commented Jun 20, 2024 •

edited

Loading

codecov-commenter commented Jun 24, 2024 •

edited by codecov bot

Loading

valentinewallace commented Jul 8, 2024

TheBlueMatt commented Jul 10, 2024

TheBlueMatt Jul 10, 2024

valentinewallace Jul 12, 2024

jkczyz Sep 4, 2024

jkczyz Sep 4, 2024

TheBlueMatt Jul 10, 2024

valentinewallace Jul 12, 2024

TheBlueMatt Jul 11, 2024

valentinewallace Jul 12, 2024

valentinewallace commented Sep 13, 2024

valentinewallace commented Sep 13, 2024

jkczyz Sep 13, 2024

TheBlueMatt left a comment

TheBlueMatt Sep 12, 2024

TheBlueMatt Sep 13, 2024

TheBlueMatt Sep 13, 2024

jkczyz Sep 16, 2024

TheBlueMatt Sep 16, 2024

TheBlueMatt Sep 13, 2024

TheBlueMatt Sep 15, 2024

TheBlueMatt Sep 15, 2024

TheBlueMatt Sep 15, 2024

TheBlueMatt Sep 15, 2024

TheBlueMatt Sep 15, 2024

Support paying static invoices #3140

Support paying static invoices #3140

Conversation

valentinewallace commented Jun 20, 2024 • edited Loading

codecov-commenter commented Jun 24, 2024 • edited by codecov bot Loading

Codecov Report

valentinewallace commented Jul 8, 2024

TheBlueMatt commented Jul 10, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valentinewallace commented Sep 13, 2024

valentinewallace commented Sep 13, 2024

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

valentinewallace commented Jun 20, 2024 •

edited

Loading

codecov-commenter commented Jun 24, 2024 •

edited by codecov bot

Loading