Feat: Support LLMs for i18n Translation + remove CrowdIn ads

[ardigan6]

Have you read the Contributing Guidelines on issues?

• I have read the Contributing Guidelines on issues.

Description

Add i18n auto-translation support using LLMs via a flexible API.

Has this been requested on Canny?

No response

Motivation

Modern LLMs are better technical translators than the average CrowdIn human.

It would make sense to fully automate the translation flow in CI, and possibly deprecate CrowdIn support mentions in the future.

API design

docusaurus --translation-source-lang=en --translation-target-langs=fr,de --translation-llm-endpoint=[openai-compatible router]

Have you tried building it?

This should be a core feature.

Self-service

• I'd be willing to contribute this feature to Docusaurus myself.

[slorber]

Docusaurus is based on file system paths.

We only document Crowdin usage for historical reasons, because Docusaurus v1 sites used it in 2017, and we wanted to help them upgrade to Docusaurus v2 (including ours and other sites at Meta). But we don't have a tight coupling to Crowdin, nor do we have any integration code for it. You don't have to use Crowdin, and we don't plan to deprecate the integration we don't even have.

The community can build a CLI to automatically translate MDX content. The only requirement is to respect the file system conventions we have.

We could be interested having this a first-class Docusaurus feature, but I'd rather see a successful community implementation first, so that we understand what works and what doesn't.

[ardigan6]

Docusaurus is based on file system paths.

We only document Crowdin usage for historical reasons, because Docusaurus v1 sites used it in 2017, and we wanted to help them upgrade to Docusaurus v2 (including ours and other sites at Meta). But we don't have a tight coupling to Crowdin, nor do we have any integration code for it. You don't have to use Crowdin, and we don't plan to deprecate the integration we don't even have.

The community can build a CLI to automatically translate MDX content. The only requirement is to respect the file system conventions we have.

We could be interested having this a first-class Docusaurus feature, but I'd rather see a successful community implementation first, so that we understand what works and what doesn't.

Yes, that was a bit unclear. I was referring to the README mentions, docs tutorial, etc. Hard to imagine >1% of docusaurus users wanting to use CrowdIn in 2025.

There are several LLM translation scripts available, e.g. https://github.com/moonrailgun/docusaurus-i18n
https://github.com/DanRoscigno/translate_changed_files

My version uses a different approach:

1. parses the mdx,
2. extracts strings only from header/table/string blocks (e.g. ignoring code),
3. replaces any code in the string with $CODEVAR[n], and then
4. chunks those strings into hash-keyed JSON ({"3d653119": "hello"}) for each supported language,
5. sends chunk to LLM endpoint for i18n, then
6. sends the output along with the original English to a second LLM to review, mark any hashes with errors, and provide an alternative translation for them, then
7. copies base files to each language and applies string replacements on top of them.

This works well, enables granular and efficient caching, and avoids many issues with more naive approaches that break mdx formatting or embedded code blocks.

Here is a simplified version of this approach for reference: https://github.com/ardigan6/docusaurus-llm-translator

[slorber]

Hard to imagine >1% of docusaurus users wanting to use CrowdIn in 2025.

We use it for Meta sites and don't plan to migrate away in the short term.

This solution is not mandatory, so if you don't like it, just don't use it. There's no need to criticize it. Adding automatic LLM translations and removing Crowdin from our docs are totally different topics.

---

This works well, enables granular and efficient caching, and avoids many issues with more naive approaches that break mdx formatting or embedded code blocks.

Here is a simplified version of this approach for reference: ardigan6/docusaurus-llm-translator

If you are already able to achieve your need with your own code, what's the value of adding this to Docusaurus core?

[ardigan6]

Hard to imagine >1% of docusaurus users wanting to use CrowdIn in 2025.

We use it for Meta sites and don't plan to migrate away in the short term.

Have you benchmarked this against the latest models? CrowdIn vendor results were uniformly inferior for the languages I speak, and of course much slower than an API call.

If you are already able to achieve your need with your own code, what's the value of adding this to Docusaurus core?

I've seen very few Docusaurus sites in the wild that enable i18n, which is a shame.

Probably this is somewhat related to the difficulty in auto-translating the content compared to a traditional CMS, where structured storage means string replacements at the field level are easy to handle safely.

Building it into core would make it simpler to reuse AST walking rather than my regex strategy, which was quick to write and works in my case but undoubtedly would fail on some docs.

[Blargian]

@ardigan6 you can take a look at our setup if you'd like https://github.com/ClickHouse/clickhouse-docs. Currently we translate from English to Chinese, Japanese and Russian.

[ardigan6]

https://github.com/ClickHouse/clickhouse-docs

@Blargian MarkdownNodeParser is a nice find, thanks. Forcing static refs (### Supported data types {#supported-data-types} etc) is also a good trick for avoiding URL breakage. Is that automatically generated and inserted somewhere?

[Blargian]

https://github.com/ClickHouse/clickhouse-docs

@Blargian MarkdownNodeParser is a nice find, thanks. Forcing static refs (### Supported data types {#supported-data-types} etc) is also a good trick for avoiding URL breakage. Is that automatically generated and inserted somewhere?

@ardigan6, we ran a script to insert them across our docs when we first introduced the change but we don't auto generate them with new additions to the docs, we just have a markdown lint rule that fails CI if people don't insert them. Adding them automatically is a nice idea for an improvement actually.

Deployment is a little messy btw. We have separate Vercel deploys for each translation and the web worker takes care of which deploy to route to based on if someone navigates /ru, /zh, /jp. Not exactly following the intended i18n approach but it works for us.

[ardigan6]

https://github.com/ClickHouse/clickhouse-docs

@Blargian MarkdownNodeParser is a nice find, thanks. Forcing static refs (### Supported data types {#supported-data-types} etc) is also a good trick for avoiding URL breakage. Is that automatically generated and inserted somewhere?

@ardigan6, we ran a script to insert them across our docs when we first introduced the change but we don't auto generate them with new additions to the docs, we just have a markdown lint rule that fails CI if people don't insert them. Adding them automatically is a nice idea for an improvement actually.

Yes, might borrow a few of these ideas for my version :) (https://github.com/ardigan6/docusaurus-llm-translator)

Deployment is a little messy btw. We have separate Vercel deploys for each translation and the web worker takes care of which deploy to route to based on if someone navigates /ru, /zh, /jp. Not exactly following the intended i18n approach but it works for us.

Surprised N deploys for every change was easier for you, what was the goal there?

[Blargian]

https://github.com/ClickHouse/clickhouse-docs

@Blargian MarkdownNodeParser is a nice find, thanks. Forcing static refs (### Supported data types {#supported-data-types} etc) is also a good trick for avoiding URL breakage. Is that automatically generated and inserted somewhere?

@ardigan6, we ran a script to insert them across our docs when we first introduced the change but we don't auto generate them with new additions to the docs, we just have a markdown lint rule that fails CI if people don't insert them. Adding them automatically is a nice idea for an improvement actually.

Yes, might borrow a few of these ideas for my version :) (https://github.com/ardigan6/docusaurus-llm-translator)

Deployment is a little messy btw. We have separate Vercel deploys for each translation and the web worker takes care of which deploy to route to based on if someone navigates /ru, /zh, /jp. Not exactly following the intended i18n approach but it works for us.

Surprised N deploys for every change was easier for you, what was the goal there?

We kick off the translations only once a month generally so it’s not deploying each time on all changes. We have a lot of pages (1300+) so deploying 4 sets of docs would increase build time a lot if we had to do it all at once. This way we keep build times short for the English docs which are frequently updated and just accept that there is some delay for the translations as a tradeoff.

The translation script only translates files with changes in any case, so we can always rerun it on an adhoc basis if there is some newly added content we really need to be translated less than monthly.