Implement semantic tokens

[jakebailey]

Fixes #1894

I am a little scared of this one. VS Code seems to ask for doc highlights, semantic tokens, diags, all in quick succession. All three of those require checkers, which means we are likely creating 3 checkers to handle these. 4 if we merge inlay hints.

[Copilot]

Pull Request Overview

This PR implements semantic token support for the TypeScript language server, enabling syntax highlighting based on semantic understanding of code (types, symbols, modifiers) rather than just syntax. This is a major editor feature that enhances the developer experience.

Key changes:

• Added semantic token generation in the language service layer with type-based classification and modifier detection
• Integrated semantic token handlers into the LSP server with support for both full-document and range-based requests
• Created extensive test infrastructure with 40+ test files covering various semantic classification scenarios

Reviewed Changes

Copilot reviewed 55 out of 55 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
internal/ls/semantictokens.go	New file implementing semantic token collection, classification, and LSP encoding logic
internal/lsp/server.go	Adds LSP handlers for semantic tokens requests and advertises semantic token capability
internal/fourslash/semantictokens.go	Test utilities for verifying semantic token output
internal/fourslash/fourslash.go	Updates test framework to support semantic token client capabilities
internal/fourslash/_scripts/convertFourslash.mts	Adds parsing support for semantic classification test cases
testdata/baselines/reference/*.jsonc	Updated test baselines showing improved behavior for rename/reference operations
internal/fourslash/tests/gen/*.go	40+ generated test files for semantic token verification
internal/fourslash/tests/manual/*.go	Manual test for semantic token classification

[DanielRosenwasser]

VS Code seems to ask for doc highlights, semantic tokens, diags, all in quick succession

Wasn't the same true for document highlights and semantic tokens?

Does it feel reasonable in a larger codebase?

[DanielRosenwasser]

Ah, but we didn't make more checkers.

[Copilot]

Pull Request Overview

Copilot reviewed 56 out of 56 changed files in this pull request and generated 3 comments.

[jakebailey]

Drafting this for now, until we can improve the checker situation IMO