:gemoji: gets an unwanted slash when converting from AST to Markdown

[vrepsys]

Hello all,

looking for some guidance with this issue I have.

The issue I'm facing is that remark-directive messes with how gemojis are rendered to from AST->Markdown.

Basically if there's a paragraph with a :gemoji:,

1. It parses into an AST with a paragraph containing a :gemoji:.(Geat, just as documented!)
2. AST with the paragraph containing :gemoji: is rendered back to markdown, it now has a backslash in front of it /:gemoji: (Not good)

Is there a reason why it gets a backslash? I understand why a paragraph with :textDirective should be escaped, but :this: is not a text directive. Just want to understand.

const ast = fromMarkdown(":rocket:", {
  extensions: [directive()],
  mdastExtensions: [directiveFromMarkdown()],
});
console.log(JSON.stringify(ast.children[0].type)); // paragraph

const md = toMarkdown(ast.children[0], { extensions: [directiveToMarkdown()] });
console.log(md); // \:rocket:

I dug a bit into mdast-util-directive code, trying to find answers.
It seems like this part might have something to do with how AST is converted to Markdown

export function directiveToMarkdown() {
  return {
    unsafe: [
      {
        character: '\r',
        inConstruct: ['leafDirectiveLabel', 'containerDirectiveLabel']
      },
      {
        character: '\n',
        inConstruct: ['leafDirectiveLabel', 'containerDirectiveLabel']
      },
      {
        before: '[^:]',
        character: ':',
        after: '[A-Za-z]',
        inConstruct: ['phrasing']
      },
      {atBreak: true, character: ':', after: ':'}
    ],
    handlers: {
      containerDirective: handleDirective,
      leafDirective: handleDirective,
      textDirective: handleDirective
    }
  }
}

Couldn't figure out which part exactly cause the slash to appear. If anyone has recommendations for how to start learning how micromark extensions work please share! I've already read the micromark README.

Sorry for long text, too late in the evening to write short and concise:)

[ChristianMurphy]

Welcome @vrepsys! 👋

Linking a related discussion, it feels like this may be similar to or the same as https://github.com/orgs/remarkjs/discussions/651

Where because of the overlap in syntax between gemoji and text directive, an escape is added by remark directive to ensure serialization does not accidentally add a directive where there should not be one.

[vrepsys]

Thanks @ChristianMurphy, useful context!

My understanding is that :this: doesn't need an escape baskslash, becaue it's not a directive. But that's probablly a side effect of escaping :real directives.

[wooorm]

Correct, but that requires a lot of context. When looking at :, we’d need to know many future characters, to find that other :, with a valid word in between. We don’t have that.
We only have the previous and next character surrounding :

[wooorm]

AST with the paragraph containing :gemoji: is rendered back to markdown, it now has a backslash in front of it /:gemoji: (Not good)

GH supports this. :sweat_smile: -> 😅, :sweat\_smile: -> 😅, \:+1: -> 👍

---

Why doesn’t your setup support this?

[wooorm]

My q is about how they serialize as well! Whtehr the brackets and braces get encoded or not!

[vrepsys]

Good morning!

it looks like the requirement for [] or {} for texts is also recent

What requirements do you have in mind? Github? Do you have a link?

I'm now wondering if parsing of \:gemoji: as a gemoji node is correct behaviour. Shouldn't the slash escape the first semicolon and cause the whole thing to render as text?

https://spec.commonmark.org/0.30/#backslash-escapes

[wooorm]

What requirements do you have in mind? Github? Do you have a link?

micromark-extension-directive changed. It used to see :a as a text directive. It no longer does.
mdast-util-directive is perhaps unaware of this change.

I'm now wondering if parsing of \:gemoji: as a gemoji node is correct behaviour. Shouldn't the slash escape the first semicolon and cause the whole thing to render as text?

You can take that up with GitHub (theoretically, because practically they don’t really work on their markdown things).
Gemoji are not an extension to markdown.
They are instead something that is post processed, transformed on the HTML tree.
At that point, escapes no longer exist.

[vrepsys]

micromark-extension-directive changed. It used to see :a as a text directive. It no longer does.
mdast-util-directive is perhaps unaware of this change.

Interesting! Do I understand correctly that if :a is no longer a directive :a: doesn't have to be escaped any longer? It's just that it hasn't been implemented in mdast-util-directive yet.

They are instead something that is post processed, transformed on the HTML tree.

Got it

[wooorm]

Perhaps. But we don’t know if :a is followed by [ or a {. If we’re not escaping :, we need to escape all [ and {s. That might also have downsides.