Parse non-syntactic operator tokens as K"Identifier" kind
#523
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Most operators are semantically just normal identifiers after parsing so get the Kind `K"Identifier"`. For example, `a + b` parses with `K"Identifier"` kind for the `+` token. As an exception, standalone syntactic ops keep their kind - they can't really be used in a sane way as identifiers or interpolated into expressions in the normal way because they have their own syntactic forms. This also helps us in `Expr` conversion where they also have their own rules for coalescing with dots, when dotted. Also introduce a new keyword `operators_as_identifiers` to the `tokenize()` API to accommodate some simple uses of this API to colour token strings by operator type, even when the operator is semantically in identifier-position.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Most operators are semantically just normal identifiers after parsing so should get the Kind
K"Identifier". For example, after this changea + bparses withK"Identifier"kind for the+token.As an exception, standalone syntactic ops keep their kind - they can't really be used in a sane way as identifiers or interpolated into expressions in the normal way because they have their own syntactic forms. This also helps us in
Exprconversion where they also have their own rules for coalescing with dots, when dotted.Also introduce a new keyword
operators_as_identifiersto thetokenize()API to accommodate some simple uses of this API to colour token strings by operator type, even when the operator is semantically in identifier-position.Fix #474 as a follow-on from #456
CC @fredrikekre