Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip verifyCompilerOptions when possible on program updates #60754

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

andrewbranch
Copy link
Member

@andrewbranch andrewbranch commented Dec 13, 2024

Related to #60633

Reusing config diagnostics lowers the update time in the linked issue scenario from ~70ms to ~50ms on my M2 Mac.

@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Dec 13, 2024
@andrewbranch
Copy link
Member Author

@typescript-bot perf test

@typescript-bot
Copy link
Collaborator

typescript-bot commented Dec 13, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

@andrewbranch
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-Unions - node (v18.15.0, x64)
Errors 34 34 ~ ~ ~ p=1.000 n=6
Symbols 62,363 62,363 ~ ~ ~ p=1.000 n=6
Types 50,395 50,395 ~ ~ ~ p=1.000 n=6
Memory used 196,145k (± 0.83%) 195,639k (± 0.96%) ~ 193,189k 196,904k p=1.000 n=6
Parse Time 1.61s (± 0.91%) 1.60s (± 1.18%) ~ 1.58s 1.62s p=0.330 n=6
Bind Time 0.88s (± 1.88%) 0.86s (± 0.94%) ~ 0.85s 0.87s p=0.133 n=6
Check Time 11.77s (± 0.46%) 11.75s (± 0.24%) ~ 11.71s 11.78s p=0.465 n=6
Emit Time 3.39s (± 3.96%) 3.29s (± 0.83%) ~ 3.26s 3.32s p=0.126 n=6
Total Time 17.65s (± 0.76%) 17.51s (± 0.27%) -0.14s (- 0.78%) 17.46s 17.58s p=0.045 n=6
angular-1 - node (v18.15.0, x64)
Errors 37 37 ~ ~ ~ p=1.000 n=6
Symbols 947,936 947,936 ~ ~ ~ p=1.000 n=6
Types 410,955 410,955 ~ ~ ~ p=1.000 n=6
Memory used 1,226,035k (± 0.00%) 1,225,995k (± 0.00%) ~ 1,225,924k 1,226,033k p=0.378 n=6
Parse Time 8.11s (± 0.91%) 8.10s (± 0.98%) ~ 7.99s 8.17s p=0.688 n=6
Bind Time 2.29s (± 1.15%) 2.27s (± 1.10%) ~ 2.25s 2.31s p=0.292 n=6
Check Time 38.10s (± 0.55%) 38.22s (± 0.45%) ~ 38.08s 38.54s p=0.230 n=6
Emit Time 18.34s (± 0.38%) 18.30s (± 0.63%) ~ 18.14s 18.42s p=0.687 n=6
Total Time 66.84s (± 0.42%) 66.88s (± 0.28%) ~ 66.63s 67.07s p=0.521 n=6
mui-docs - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 2,448,065 2,448,065 ~ ~ ~ p=1.000 n=6
Types 896,131 896,131 ~ ~ ~ p=1.000 n=6
Memory used 2,320,429k (± 0.00%) 2,320,454k (± 0.00%) ~ 2,320,363k 2,320,554k p=0.936 n=6
Parse Time 11.32s (± 0.23%) 11.33s (± 0.56%) ~ 11.27s 11.45s p=0.747 n=6
Bind Time 2.66s (± 0.19%) 2.65s (± 0.46%) ~ 2.63s 2.66s p=0.070 n=6
Check Time 87.80s (± 1.22%) 87.77s (± 1.41%) ~ 86.90s 90.21s p=0.936 n=6
Emit Time 0.35s (± 2.55%) 0.34s (± 3.01%) ~ 0.33s 0.35s p=0.344 n=6
Total Time 102.12s (± 1.03%) 102.10s (± 1.19%) ~ 101.23s 104.49s p=0.810 n=6
self-build-src - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,331 1,225,378 +47 (+ 0.00%) ~ ~ p=0.001 n=6
Types 266,569 266,579 +10 (+ 0.00%) ~ ~ p=0.001 n=6
Memory used 3,084,905k (± 0.02%) 2,963,594k (±10.02%) ~ 2,356,552k 3,085,707k p=0.936 n=6
Parse Time 6.78s (± 0.36%) 6.77s (± 1.56%) ~ 6.58s 6.87s p=0.575 n=6
Bind Time 2.14s (± 1.46%) 2.13s (± 1.14%) ~ 2.11s 2.17s p=0.809 n=6
Check Time 42.72s (± 0.40%) 42.76s (± 0.21%) ~ 42.63s 42.86s p=0.423 n=6
Emit Time 3.44s (± 1.53%) 3.44s (± 2.90%) ~ 3.29s 3.57s p=0.748 n=6
Total Time 55.07s (± 0.26%) 55.11s (± 0.23%) ~ 54.97s 55.29s p=0.575 n=6
self-build-src-public-api - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 1,225,331 1,225,378 +47 (+ 0.00%) ~ ~ p=0.001 n=6
Types 266,569 266,579 +10 (+ 0.00%) ~ ~ p=0.001 n=6
Memory used 3,029,482k (± 9.76%) 2,787,116k (±14.26%) ~ 2,423,383k 3,150,646k p=0.298 n=6
Parse Time 6.98s (± 1.54%) 6.93s (± 2.26%) ~ 6.76s 7.13s p=0.810 n=6
Bind Time 2.15s (± 1.64%) 2.16s (± 2.33%) ~ 2.11s 2.25s p=0.809 n=6
Check Time 42.93s (± 0.82%) 42.68s (± 0.28%) ~ 42.56s 42.89s p=0.093 n=6
Emit Time 3.47s (± 2.14%) 3.52s (± 3.45%) ~ 3.40s 3.69s p=0.689 n=6
Total Time 55.53s (± 0.77%) 55.30s (± 0.32%) ~ 55.01s 55.55s p=0.128 n=6
self-compiler - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 262,270 262,284 +14 (+ 0.01%) ~ ~ p=0.001 n=6
Types 106,628 106,635 +7 (+ 0.01%) ~ ~ p=0.001 n=6
Memory used 439,885k (± 0.01%) 439,918k (± 0.01%) ~ 439,850k 440,013k p=0.575 n=6
Parse Time 3.54s (± 0.49%) 3.54s (± 1.25%) ~ 3.48s 3.60s p=0.936 n=6
Bind Time 1.32s (± 0.31%) 1.32s (± 0.79%) ~ 1.31s 1.34s p=0.528 n=6
Check Time 18.93s (± 0.33%) 19.01s (± 0.45%) ~ 18.90s 19.12s p=0.127 n=6
Emit Time 1.53s (± 1.29%) 1.53s (± 0.64%) ~ 1.52s 1.54s p=0.802 n=6
Total Time 25.32s (± 0.29%) 25.41s (± 0.42%) ~ 25.24s 25.52s p=0.199 n=6
ts-pre-modules - node (v18.15.0, x64)
Errors 70 70 ~ ~ ~ p=1.000 n=6
Symbols 226,062 226,062 ~ ~ ~ p=1.000 n=6
Types 94,488 94,488 ~ ~ ~ p=1.000 n=6
Memory used 371,625k (± 0.04%) 371,574k (± 0.01%) ~ 371,532k 371,638k p=0.936 n=6
Parse Time 2.89s (± 0.56%) 2.90s (± 1.19%) ~ 2.85s 2.95s p=0.686 n=6
Bind Time 1.59s (± 1.23%) 1.59s (± 0.40%) ~ 1.58s 1.60s p=1.000 n=6
Check Time 16.49s (± 0.26%) 16.43s (± 0.29%) ~ 16.37s 16.48s p=0.065 n=6
Emit Time 0.00s (±244.70%) 0.00s ~ ~ ~ p=0.405 n=6
Total Time 20.98s (± 0.21%) 20.92s (± 0.23%) ~ 20.83s 20.97s p=0.077 n=6
vscode - node (v18.15.0, x64)
Errors 3 3 ~ ~ ~ p=1.000 n=6
Symbols 3,215,020 3,215,020 ~ ~ ~ p=1.000 n=6
Types 1,106,063 1,106,063 ~ ~ ~ p=1.000 n=6
Memory used 3,281,141k (± 0.01%) 3,280,975k (± 0.01%) ~ 3,280,622k 3,281,467k p=0.810 n=6
Parse Time 14.20s (± 0.51%) 14.15s (± 0.61%) ~ 14.04s 14.29s p=0.196 n=6
Bind Time 4.70s (± 2.74%) 4.57s (± 2.41%) ~ 4.50s 4.79s p=0.107 n=6
Check Time 86.80s (± 0.36%) 87.52s (± 1.18%) ~ 86.76s 89.55s p=0.093 n=6
Emit Time 28.29s (± 2.62%) 28.36s (± 2.14%) ~ 27.50s 29.12s p=1.000 n=6
Total Time 133.99s (± 0.77%) 134.59s (± 0.68%) ~ 133.45s 135.95s p=0.575 n=6
webpack - node (v18.15.0, x64)
Errors 0 0 ~ ~ ~ p=1.000 n=6
Symbols 288,747 288,747 ~ ~ ~ p=1.000 n=6
Types 117,158 117,158 ~ ~ ~ p=1.000 n=6
Memory used 441,093k (± 0.03%) 440,960k (± 0.03%) ~ 440,793k 441,175k p=0.128 n=6
Parse Time 4.10s (± 0.95%) 4.10s (± 1.32%) ~ 4.03s 4.16s p=0.936 n=6
Bind Time 1.77s (± 1.31%) 1.78s (± 0.84%) ~ 1.77s 1.81s p=0.292 n=6
Check Time 18.92s (± 0.28%) 18.85s (± 0.34%) ~ 18.78s 18.95s p=0.108 n=6
Emit Time 0.00s 0.00s (±244.70%) ~ 0.00s 0.01s p=0.405 n=6
Total Time 24.79s (± 0.32%) 24.74s (± 0.33%) ~ 24.61s 24.82s p=0.378 n=6
xstate-main - node (v18.15.0, x64)
Errors 5 5 ~ ~ ~ p=1.000 n=6
Symbols 552,233 552,233 ~ ~ ~ p=1.000 n=6
Types 184,971 184,971 ~ ~ ~ p=1.000 n=6
Memory used 492,247k (± 0.01%) 492,234k (± 0.01%) ~ 492,205k 492,329k p=1.000 n=6
Parse Time 2.76s 2.76s (± 0.15%) ~ 2.75s 2.76s p=0.405 n=6
Bind Time 0.96s 0.96s ~ ~ ~ p=1.000 n=6
Check Time 16.18s (± 0.26%) 16.18s (± 0.28%) ~ 16.11s 16.23s p=1.000 n=6
Emit Time 0.00s 0.00s ~ ~ ~ p=1.000 n=6
Total Time 19.89s (± 0.22%) 19.89s (± 0.25%) ~ 19.82s 19.95s p=1.000 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Compiler-Unions - node (v18.15.0, x64)
  • angular-1 - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-build-src-public-api - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • ts-pre-modules - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate-main - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,316ms (± 0.28%) 2,314ms (± 0.28%) ~ 2,305ms 2,322ms p=0.573 n=6
Req 2 - geterr 5,290ms (± 0.62%) 5,304ms (± 0.35%) ~ 5,284ms 5,337ms p=0.378 n=6
Req 3 - references 262ms (± 1.50%) 266ms (± 1.27%) ~ 262ms 269ms p=0.061 n=6
Req 4 - navto 227ms (± 0.78%) 226ms (± 0.43%) ~ 225ms 228ms p=0.799 n=6
Req 5 - completionInfo count 1,357 1,357 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 80ms (± 7.94%) 76ms (± 0.68%) ~ 75ms 76ms p=0.142 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,420ms (± 1.89%) 2,444ms (± 1.32%) ~ 2,393ms 2,470ms p=0.378 n=6
Req 2 - geterr 4,018ms (± 0.58%) 4,030ms (± 0.42%) ~ 4,009ms 4,051ms p=0.378 n=6
Req 3 - references 281ms (± 0.27%) 280ms (± 1.21%) ~ 273ms 282ms p=0.454 n=6
Req 4 - navto 227ms (± 0.18%) 227ms (± 0.18%) ~ 226ms 227ms p=0.218 n=6
Req 5 - completionInfo count 1,519 1,519 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 86ms (± 2.73%) 85ms (± 4.47%) ~ 77ms 87ms p=0.451 n=6
xstate-main-1-tsserver - node (v18.15.0, x64)
Req 1 - updateOpen 5,276ms (± 0.29%) 5,275ms (± 0.27%) ~ 5,250ms 5,293ms p=0.809 n=6
Req 2 - geterr 1,150ms (± 1.45%) 1,154ms (± 1.24%) ~ 1,139ms 1,175ms p=0.630 n=6
Req 3 - references 83ms (± 0.62%) 81ms (± 3.77%) ~ 75ms 83ms p=0.247 n=6
Req 4 - navto 450ms (± 0.51%) 449ms (± 0.48%) ~ 447ms 453ms p=0.566 n=6
Req 5 - completionInfo count 3,450 3,450 ~ ~ ~ p=1.000 n=6
Req 5 - completionInfo 848ms (± 2.03%) 842ms (± 1.01%) ~ 832ms 854ms p=0.810 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstate-main-1-tsserver - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 160.01ms (± 0.22%) 160.16ms (± 0.21%) +0.15ms (+ 0.09%) 158.98ms 164.22ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 233.55ms (± 0.16%) 233.40ms (± 0.16%) -0.16ms (- 0.07%) 232.08ms 237.28ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 229.47ms (± 0.15%) 229.46ms (± 0.16%) ~ 227.92ms 235.51ms p=0.534 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 228.44ms (± 0.16%) 228.40ms (± 0.19%) -0.04ms (- 0.02%) 226.77ms 235.45ms p=0.032 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@@ -2722,6 +2735,14 @@ export function createProgram(rootNamesOrOptions: readonly string[] | CreateProg
filesByName.set(path, filesByName.get(oldFile.path));
});

if (
oldOptions.configFile && oldOptions.configFile === options.configFile ||
!oldOptions.configFile && !options.configFile && !optionsHaveChanges(oldOptions, options, optionDeclarations)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think you need to check that the program's file set are the same as well, because a new or deleted file can invalidate/introduce some errors. Maybe it's done elsewhere, but it would be good to add a test for that.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Early returns above handle that (we wouldn’t be at StructureIsReused.Completely if that happened)

@DanielRosenwasser
Copy link
Member

@typescript-bot pack this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Dec 13, 2024

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
pack this ✅ Started ✅ Results

@typescript-bot
Copy link
Collaborator

typescript-bot commented Dec 13, 2024

Hey @DanielRosenwasser, I've packed this into an installable tgz. You can install it for testing by referencing it in your package.json like so:

{
    "devDependencies": {
        "typescript": "https://typescript.visualstudio.com/cf7ac146-d525-443c-b23c-0d58337efebc/_apis/build/builds/164389/artifacts?artifactName=tgz&fileId=B0A5033E72FA516769924862354F3C9291EF74999415900BE634CCF66DFD3DCF02&fileName=/typescript-5.8.0-insiders.20241213.tgz"
    }
}

and then running npm install.


There is also a playground for this build and an npm module you can use via "typescript": "npm:@typescript-deploys/[email protected]".;

@andrewbranch andrewbranch changed the title updateGraph perf experiments Skip verifyCompilerOptions when possible on program updates Dec 18, 2024
@andrewbranch andrewbranch marked this pull request as ready for review December 18, 2024 17:40
Comment on lines +2590 to +2591
const isConfigIdentical = oldOptions.configFile && oldOptions.configFile === options.configFile
|| !oldOptions.configFile && !options.configFile && !optionsHaveChanges(oldOptions, options, optionDeclarations);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wanted to suggest

Suggested change
const isConfigIdentical = oldOptions.configFile && oldOptions.configFile === options.configFile
|| !oldOptions.configFile && !options.configFile && !optionsHaveChanges(oldOptions, options, optionDeclarations);
// Reuse if both have an identical config file,
// or neither have a config but the options have changed.
const isConfigIdentical = oldOptions.configFile === options.configFile && (
options.configFile || !optionsHaveChanges(oldOptions, options, optionDeclarations))

But now the comment is really explaining what the current code is more clearly doing. That said, when do you end up in a situation where neither has a configFile? Inferred projects?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, possibly—I was also wondering about tsc --watch without -p but I honestly don’t know for sure

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants