v11.2.0 post-mortem

[kyliau]

What happened?

v11.2.0 released on Friday February 12, 2021 broke users who are using the Ivy-native language service.

Who was affected?

This affected users who are using the extension in Ivy mode.

How long was the incident?

We received user reports over the weekend, but since Monday Feb 15 was holiday, the issue was not noticed until Tuesday Feb 16. The broken extension was available in the vscode marketplace from Feb 10 10:45am PT till Feb 16 1pm PT.

Immediate response

A community member has already identified the bug and submitted a PR, so we quickly pushed out a fix in v11.2.1.

What made this incident worse?

The v11.2.0 release contains a feature commit to prompt users to upgrade to Ivy mode, and I believe many users did.
However, immediately upon opting-in, their extension stopped working.

How did it happen?

• Due to a bug in vsce, the dependencies are not pinned for the production build. This is because we use npm to install dependencies in production, but we use yarn to install dependencies in dev.
• Feb 10: TS 4.1.5 was released, and it contains a commit that disables loading of @angular/language-service/bundles/ivy due to security concerns. It only allows package "root names".
• Feb 12: When the production bundle was built, it picked up the new TS version that's different from the pinned version.
• The release instructions explicitly call out the need for manual testing of the production build, but I did the testing in dev, so this regression went unnoticed.

How to prevent this from happening again?

• Make the release process more vigorous, follow the instructions strictly.
• This will make caretaking more time-consuming, but we could share the rotation with other team members.
• Do not release on Friday, especially when there's a long weekend ahead.
• Remove the need for manual testing in the release process.
• Pin all dependencies to make sure RenovateBot picks up latest versions.
• Avoid two sets of dependencies (yarn vs npm).

Long term fix

Although #1111 fixed the immediate problem, the long term solution should be in @angular/language-service.
We should have a single entry point, but make use of the plugin config to load View Engine / Ivy.

[angular-automatic-lock-bot]

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

_{This action has been performed automatically by a bot.}