Error expanding macro recursively at caret: $crate cannot come from SyntaxContextId::ROOT

[heaths]

When trying to expand a proc_macro, most often the rust-analyzer errs with:

What's odd is that it doesn't always happen. Sometimes I can rebuild dependencies in our workspace, restart the LSP server, and it'll work. I also don't use $crate myself, though #[tokio::test], assert_eq!(), and some other macros may that end up getting expanded recursively.

This most often happens when I make a change to my crate e.g., minor bug fix or small behavior change as I tweak what it does for this in-development feature, and then try to expand the macro. rust-analyzer is rebuilding dependencies, but I've suspected it's a cached build output issue somehow.

rust-analyzer version: (eg. output of "rust-analyzer: Show RA Version" command, accessible in VSCode via Ctrl/⌘+Shift+P)

0.3.2212-standalone

rustc version: (eg. output of rustc -V)

rustc 1.80.0 (051478957 2024-07-21) (pinned via rust-toolchain.toml)

editor or extension: (eg. VSCode, Vim, Emacs, etc. For VSCode users, specify your extension version; for users of other editors, provide the distribution if applicable)

VSCode rust-lang.rust-analyzer extension version v0.3.2212

relevant settings: (eg. client settings, or environment variables like CARGO, RUSTC, RUSTUP_HOME or CARGO_HOME)

Only a few relative things come to mind:

• rust-toolchain.toml pins channel to 1.80.0
• .vscode/settings.json sets "rust-analyzer.cargo.features": "all" and "rust-analyzer.check.command": "clippy".

repository link (if public, optional): (eg. rust-analyzer)

• proc_macro: https://github.com/heaths/azure-sdk-for-rust/blob/09c9a45b1a267f304d34a55567e2935633f60cbe/sdk/core/azure_core_test_macros/src/test.rs
• use case: https://github.com/heaths/azure-sdk-for-rust/blob/09c9a45b1a267f304d34a55567e2935633f60cbe/sdk/cosmos/azure_data_cosmos/tests/cosmos_containers.rs#L19

code snippet to reproduce:

See links directly above.

[heaths]

See #18163 for possibly more context.

[ChayimFriedman2]

Is there a backtrace in the logs?

[heaths]

Found it in the Output pane for the Rust Analyzer LSP:

thread 'Worker' panicked at crates/hir-expand/src/prettify_macro_expansion_.rs:30:41:
`$crate` cannot come from `SyntaxContextId::ROOT`
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::option::expect_failed
   3: hir_expand::prettify_macro_expansion_::prettify_macro_expansion::{{closure}}
   4: syntax_bridge::prettify_macro_expansion::prettify_macro_expansion
   5: hir_expand::prettify_macro_expansion_::prettify_macro_expansion
   6: ide::expand_macro::format
   7: ide::expand_macro::expand_macro
   8: ra_salsa::Cancelled::catch
   9: rust_analyzer::handlers::request::handle_expand_macro
  10: core::ops::function::FnOnce::call_once{{vtable.shim}}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Want me to get a full backtrace? Seems, though, it started in frame 3 or 4.

[stevenlele]

@ChayimFriedman2 This is because there's a bug in expanding recursive attribute macro, breaking $crate resolution.

rust-analyzer/crates/ide/src/expand_macro.rs

Lines 155 to 187 in 9b2e72c

	fn expand(
	sema: &Semantics<'_, RootDatabase>,
	expanded: SyntaxNode,
	result_span_map: &mut SpanMap<SyntaxContextId>,
	mut offset_in_original_node: i32,
	) -> SyntaxNode {
	let children = expanded.descendants().filter_map(ast::Item::cast);
	let mut replacements = Vec::new();
	for child in children {
	if let Some(new_node) = expand_macro_recur(
	sema,
	&child,
	result_span_map,
	TextSize::new(
	(offset_in_original_node + (u32::from(child.syntax().text_range().start()) as i32))
	as u32,
	),
	) {
	offset_in_original_node = offset_in_original_node
	+ (u32::from(new_node.text_range().len()) as i32)
	- (u32::from(child.syntax().text_range().len()) as i32);
	// check if the whole original syntax is replaced
	if expanded == *child.syntax() {
	return new_node;
	}
	replacements.push((child, new_node));
	}
	}
	replacements.into_iter().rev().for_each(|(old, new)| ted::replace(old.syntax(), new));
	expanded
	}

When expanded itself is still an attribute macro invocation, only children[0] (itself) should be expanded. children[1..] are actually its subtrees. However, in this code the subtrees are expanded too after the correct expansions. The result was not broken since the subtrees became orphaned syntax nodes, but the span_map you added for $crate resolution is broken due to the wrong calculations - the wrong expansions with wrong offset numbers overwrites the correct information. I think this can be fixed by checking if expanded itself is another attribute macro that can be expanded again.

A bit more about my debugging process: I found that the span_map, if computed correctly, should be in order with iterated tokens:

rust-analyzer/crates/hir-expand/src/prettify_macro_expansion_.rs

Lines 12 to 25 in 9b2e72c

	pub fn prettify_macro_expansion(
	db: &dyn ExpandDatabase,
	syn: SyntaxNode,
	span_map: &ExpansionSpanMap,
	target_crate_id: CrateId,
	) -> SyntaxNode {
	// Because `syntax_bridge::prettify_macro_expansion::prettify_macro_expansion()` clones subtree for `syn`,
	// that means it will be offsetted to the beginning.
	let span_offset = syn.text_range().start();
	let crate_graph = db.crate_graph();
	let target_crate = &crate_graph[target_crate_id];
	let mut syntax_ctx_id_to_dollar_crate_replacement = FxHashMap::default();
	syntax_bridge::prettify_macro_expansion::prettify_macro_expansion(syn, &mut |dollar_crate| {
	let ctx = span_map.span_at(dollar_crate.text_range().start() + span_offset).ctx;

rust-analyzer/crates/syntax-bridge/src/prettify_macro_expansion.rs

Lines 61 to 64 in 9b2e72c

	if token.kind() == SyntaxKind::IDENT && token.text() == "$crate" {
	dollar_crate_replacements.push((token.clone(), dollar_crate_replacement(&token)));
	}
	let tok = &token;

// in snippet 1
let mut it = span_map.iter();

// in snippet 2
assert_eq!(token.text_range().end(), it.next().unwrap().0)

Adding a sanity check here reveals that the span_map has corrupted data, with some mismatching tokens. The duplicated information means there might be some design issue, but for safety I think you can add this check too, for now.

Here's a minimal reproducing snippet:

// macros
#[proc_macro_attribute]
pub fn foo(args: TokenStream, input: TokenStream) -> TokenStream {
    let func = parse_macro_input!(input as syn::ItemFn);
    let msg = parse_macro_input!(args as syn::LitStr);
    let ident = &func.sig.ident;
    quote! {
        #[bar]
        fn #ident() {
            log::info!(#msg);
            log::info!(#msg);
            log::info!(#msg);
            log::info!(#msg);
        }
    }.into()
}

#[proc_macro_attribute]
pub fn bar(_args: TokenStream, input: TokenStream) -> TokenStream {
    let func = parse_macro_input!(input as syn::ItemFn);
    let ident = &func.sig.ident;
    quote! {
        fn #ident() {
            panic!("bar");
            panic!("bar");
            panic!("bar");
            panic!("bar");
            panic!("bar");
            panic!("bar");
        }
    }.into()
}

// invocation
extern crate log;
use reproduce::{foo, bar};

#[foo("foo")]
fn wrong_crate() {}

#[foo("fooooooooooooooooooooooooooooooooooooooooooooooo")]
fn would_panic() {}

// Recursive expansion of foo macro
// =================================

fn wrong_crate() {
    {
        core::panicking::panic_fmt(core::const_format_args!("bar"));
    };
    {
        core::panicking::panic_fmt(core::const_format_args!("bar"));
    };
    {
        core::panicking::panic_fmt(core::const_format_args!("bar"));
    };
    {
        core::panicking::panic_fmt(core::const_format_args!("bar"));
    };
    {
        core::panicking::panic_fmt(core::const_format_args!("bar"));
    };
    {
        log::panicking::panic_fmt(log::const_format_args!("bar"));  // <- `$crate` becomes `log`
    };
}

// expand `would_panic`
thread 'Worker' panicked at crates/hir-expand/src/prettify_macro_expansion_.rs:30:41:
`$crate` cannot come from `SyntaxContextId::ROOT`
stack backtrace:
   0: rust_begin_unwind
   1: core::panicking::panic_fmt
   2: core::option::expect_failed
   3: hir_expand::prettify_macro_expansion_::prettify_macro_expansion::{{closure}}
   4: syntax_bridge::prettify_macro_expansion::prettify_macro_expansion
   5: hir_expand::prettify_macro_expansion_::prettify_macro_expansion
   6: ide::expand_macro::format
   7: ide::expand_macro::expand_macro
   8: ra_salsa::Cancelled::catch
   9: rust_analyzer::handlers::request::handle_expand_macro
  10: core::ops::function::FnOnce::call_once{{vtable.shim}}
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.