fix(transformers): Ignore empty lines on notation transformer range

[fuma-nama]

Description

Start counting from next line of the comment if the current line is empty.

For example:

// [!code highlighted:2]
highlighted
highlighted

highlighted // [!code highlighted:2]
highlighted

Linked Issues

#619

Additional context

I made some changes to the design of createCommentNotationTransformer so it matches the comment token without a given regex, I've documented it in case for other contributors who doesn't understand it well.

note: forgot to reset local repo after the last PR was merged, apologies for the messy commit history.

[netlify]

❌ Deploy Preview for shiki-matsu failed.

Name	Link
🔨 Latest commit	d994692
🔍 Latest deploy log	https://app.netlify.com/sites/shiki-matsu/deploys/66daa107edebba00088c46e9

[netlify]

❌ Deploy Preview for shiki-next failed.

Name	Link
🔨 Latest commit	d994692
🔍 Latest deploy log	https://app.netlify.com/sites/shiki-next/deploys/66daa10771cf290008252980

[fuma-nama]

Also I realized this is a good opportunity to improve the current behaviors of notation transformers.

Like:

Normal

console.log("hello") // [!code xxx] original comment

Multiple transformers

console.log("hello") // [!code highlight] [!code world:hello] original comment

(maybe we can support the previous // original comment // [!code highlight] // [!code world:hello] syntax?)

impact next line if the current line is empty (this PR)

// [!code highlight] [!code world:hello]
console.log("hello") // original comment

keep current line

// [!code highlight] [!code current]
console.log("hello") // [!code highlight]

[!code current] will keep and highlight the empty line. I think it would be required once this PR gets merged.

it requires some extra work but won't cause a break change except the current proposal of "Ignoring empty lines"

[fuma-nama]

@antfu I'm fixing some edge cases and I found one of the test cases:

function hello(indentSize, type) {
  if (indentSize === 4 && type !== 'tab') {
    	console.log('Each next indentation will increase on 4 spaces'); // [!code error] // [!code focus]
  }
}

gives the mismatch:

I thought it was the problem of my changes, but it seems like the original snapshot was wrong.
The original snapshot didn't apply the error transformer, and my fix accidentally fixed it.

I'll commit the changes for now (it's probably a breaking change, although docs didn't mention that explicitly)

[antfu]

I am not sure if we should hard-code the regex, this makes the code less flexiable

[fuma-nama]

Hmm do you mean to allow passing the entire regex, including the part for matching comments? (like the previous code)

I assume createCommentNotationTransformer is for matching comment notations, it sounds reasonable to me to add the regex for matching comment inside the utility. I can revert it if you wanted

[antfu]

If the second arg text is coming from the match, why do we need to change it from match to text? Doesn't it make the utils less flexiable?

Instead of making a new function as breaking changes, couldn't we just allow onMatch to return a string instead of boolean?

[fuma-nama]

the new match here refers to the entire comment (// [!code] highlight), but we also want to remove the comment if the comment content becomes empty, e.g. // .

so we pass the text instead ([!code] highlight), the transformer replaces comment content, which allows us to detect if the new text is empty. This can also improve the flexibility, and remove duplicated regex for matching code comments.

As you see, the createCommentNotationTransformer now has the algorithm for matching comments. We can improve it to support other comment syntaxes.

For the problem about breaking change, I think we could also create a wrapper over the original function, would like to hear your opinions about this.

[fuma-nama]

I'm doing some experiments on notation transformers, and I found it's possible to redesign the current algorithm of notation transformers to support JSX comments e.g. #770, and optimize performance by reading only the last few tokens of each line.
Parsed comments can be cached per codeblock so chaining multiple notation transformers can lose less performance.

I putted these changes here, but this introduces breaking change as some backward incompatible changes are made, including "Ignore empty lines on notation transformer range" (the aim of this PR). I'm thinking to port the changes to a PR, if you're up to it.

[antfu]

Feel free to always create the PR (if you already work on it), so we can discuss about the implementations.

Sorry I was taking a long day offs, need to catch up and review a lot things