Implement better error recovery for parsing of separated syntax lists

[alexrp]

celerity/src/syntax/LanguageParser.cs

Lines 209 to 265 in 53f45a6

	private (ImmutableArray<T>.Builder Elements, ImmutableArray<SyntaxToken>.Builder Separators) ParseSeparatedList<T>(
	Func<LanguageParser, T> parser,
	SyntaxTokenKind separator,
	SyntaxTokenKind closer,
	bool allowEmpty,
	bool allowTrailing)
	where T : SyntaxNode
	{
	// TODO: The way we parse a parameter list (and other similar syntax nodes) causes the parser to misinterpret
	// the entire function body for some invalid inputs. We need to do better here.
	var result = SeparatedBuilder<T>();
	var (elems, seps) = result;
	bool NextIsRelevant()
	{
	return Peek1() is { IsEndOfInput: false } next && next.Kind != closer;
	}
	if (!allowTrailing)
	{
	if (allowEmpty && !NextIsRelevant())
	return result;
	elems.Add(parser(this));
	while (Optional(separator) is { } sep)
	{
	seps.Add(sep);
	elems.Add(parser(this));
	}
	return result;
	}
	if (!allowEmpty)
	{
	elems.Add(parser(this));
	if (Optional(separator) is not { } sep)
	return result;
	seps.Add(sep);
	}
	while (NextIsRelevant())
	{
	elems.Add(parser(this));
	if (Optional(separator) is not { } sep2)
	break;
	seps.Add(sep2);
	}
	return result;
	}

[alexrp]

To solve this, we will likely need a more general mechanism for dealing with skipped tokens. One approach might be to decay skipped tokens to text and attach them as trivia to another token (whether real or IsMissing).

[alexrp]

One approach might be to decay skipped tokens to text and attach them as trivia to another token (whether real or IsMissing).

This is almost certainly the way we'll go.

Note to future self: There's some code that assumes that SyntaxToken.IsMissing implies SyntaxToken.FullSpan.IsEmpty, which would no longer hold if such a token has skipped trivia attached. This will need to be fixed.

[alexrp]

Support for skipped token trivia has been added. This situation can be improved now.

[alexrp]

For reference: https://github.com/dotnet/roslyn/blob/7241b8522990905c491dee920d32dc09c345920d/src/Compilers/CSharp/Portable/Parser/LanguageParser.cs#L13631