[CodeSever] liveness and readiness probe times out due extension installation

[harshad16]

There is a problem with this approach and current Notebooks when installing more than just these few extensions at runtime. The defaults for liveness and readiness probe times come from odh-dashboard, I think, when assembling the notebook CR.

          name: code-server-vs
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /notebook/dstst-analytix-fraud/code-server-vs/api
              port: notebook-port
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 5
            successThreshold: 1
            timeoutSeconds: 1

Because installing extensions takes some time at the code-server-vs/api health check endpoint is not up and running until after all extensions install and the nginx http server is running, the container keeps re-starting without more than 5 or 6 extensions ever installed.

I solved this by patching the notebook CR to have a large initialDelaySeconds of 100, so then code server started.

What do you think? @guimou do you have an idea of how to circumvent this problem of premature container restart due to failing liveness probes when installing more than, say, 6 or 7 extensions?

Originally posted by @shalberd in #347 (comment)

[harshad16]

This is definitely an issue if the probes can't keep up. Several solutions:
- Can we start code-server and then only install the extensions?
- Right now the probe calls are forwarded by NGINX to code-server. We could
modify this to make NGINX answer directly. But it defeats the purpose of
the probes...

Honestly, I would not install any extensions. As they are persisted anyway,
I would let the users install what they want, when they want. They have to
do it only once, so... Plus the extensions installed now are related mostly
to Jupyter. Who says I want to use notebooks in VSCode? Personally I don't.
If I create a VSCode workbench, it's to do standard Python or JavaScript
coding, so...

Reference: #347 (comment)

[harshad16]

About not installing any extensions and about the whole Jupyter stuff in general in VS Code:

yes, though some things like material theme and other GUI things like German language pack are quite useful actually.

about not installing any extensions and letting users install them:

that is a good solution in non-airgapped environments. But in environments like ours, with a proxy not allowing connections to marketplaces, pypi.org and so on, this command-based install of extensions by providing them in the file system is the only way to go. I guess @atheo89 also had some restricted network customers in her automotive environment south of the Alps.

Here's some example ;-)

    code-server --install-extension ${SCRIPT_DIR}/utils/steoates.autoimport-1.5.3.vsix
    code-server --install-extension ${SCRIPT_DIR}/utils/njpwerner.autodocstring-0.6.1.vsix
    code-server --install-extension ${SCRIPT_DIR}/utils/aaron-bond.better-comments-3.0.2.vsix
    code-server --install-extension ${SCRIPT_DIR}/utils/alefragnani.Bookmarks-13.0.1.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/Catppuccin.catppuccin-vsc-3.10.1.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/formulahendry.code-runner-0.12.1.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/streetsidesoftware.code-spell-checker-3.0.0.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/streetsidesoftware.code-spell-checker-german-2.3.1.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/MS-CEINTL.vscode-language-pack-de-1.80.0.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/davidlday.languagetool-linter-0.20.0.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/PKief.material-icon-theme-4.33.0.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/ionutvmi.path-autocomplete-1.25.0.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/alefragnani.project-manager-12.0.1.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/KevinRose.vsc-python-indent-1.18.0.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/donjayamanne.python-environment-manager-1.0.4.vsix
    code-server --install-extension ${SCRIPT_DIR}/utils/hbenl.test-adapter-converter-0.1.6.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/hbenl.vscode-test-explorer-2.21.1.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/LittleFoxTeam.vscode-python-test-adapter-0.8.2.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/njqdev.vscode-python-typehint-1.4.0.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/ms-python.black-formatter-2023.6.0.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/wmaurer.change-case-1.0.0.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/mgesbert.python-path-0.0.14.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/dongli.python-preview-0.0.4.vsix
code-server --install-extension ${SCRIPT_DIR}/utils/Semgrep.semgrep-1.6.3.vsix

I wonder if there is a better way than doing the code-server --install-extension at runtime.
Ideally, this would be done at build time and then perhaps only copied to the .local user app root at runtime ...

Reference: #347 (comment)

[harshad16]

Sure. But then it's something that has to be fully customized to a specific
use case/environment. We can't impose all sorts of extensions, even a few,
to everyone just to cover this specific case.
In this scenario, one would only have to create a custom image deriving
from this one by adding the needed extensions in the utils folder.

Reference: #347 (comment)

[harshad16]

Yes, this is what I did on contrib workbench images builder. I added a custom ide snippet for my vscode and used the logic here, with all the extensions I need in the utils folder.

We can't impose all sorts of extensions, even a few, to everyone just to cover this specific case.

But we could make this a special case or extended snippet, i.e. Code Server Airgapped Extensions, of Code Server in contrib workbench-images builder.

However, the current approach here to install extensions fails due to the imposed liveness probe (initialDelaySeconds: 10) that I had to manually patch to be waiting 100s initially so the extension install can run through.

Since liveness probe is something that is statically created by odh dashboard, this is not very good.

Best way to install extensions would be with an init container mechanism, but we don't have that option with Notebook CRs here, I think. Another users in the comments below also tried to bake in the extensions at build time, I'll see what I can do. I don't want them installed at runtime in the entrypoint script, due to the liveness probe timeout problem I described.

Best way would be to have an init Container in the pod that runs before
the main pod and installs the extensions.

https://github.com/coder/code-server/discussions/4778#discussioncomment-2052391

https://github.com/coder/code-server/blob/439f27c5685d3aeba158269d162f1bf800482295/docs/helm.md#extra-containers

Since that is not possible, I'm just creating a dedicated extensions (installed extensions) dir at /opt/app-root/share/codeserver/extensions outside the mounted volume context in the Containerfile.

I then install the extensions in the Containerfile instead of the entrypoint script, with an --extensions-dir /opt/app-root/share/codeserver/extensions additional argument. Then, in the code-server run command in run-code-server.sh, I also add an additional argument --extensions-dir /opt/app-root/share/codeserver/extensions \

This works, of course, also having the usual directory permissions fixes for the new directory. Extensions are now baked in for airgapped.

Reference: #347 (comment)

[shalberd]

@harshad16 @guimou @atheo89 I got it working with installing the vsx files from utils not in the entrypoint script, but in the containerfile, with an additonal argument --extensions-dir and persisting the installed extensions to a non-mounted, fixed directory. Custom image for your notebooks would be a way to go potentially, it was easy to accomplish with Guillome's image builder on odh contrib, just a new modified snippet, essentially.

Since the container build happens on a machine with Internet access, it's a breeze, i.e. no explicit install of dependent extensions like jupyter notebooks cell renderer necessary, that happens automatically.

ARG CODESERVER_VERSION=v4.16.1
ARG CODESERVER_EXTENSIONS_DIR=/opt/app-root/share/code-server/extensions

USER 0

WORKDIR /opt/app-root/bin

RUN yum install -y "https://github.com/coder/code-server/releases/download/${CODESERVER_VERSION}/code-server-${CODESERVER_VERSION/v/}-amd64.rpm" && \
    yum -y clean all --enablerepo='*'

COPY --chown=1001:0 utils utils/

RUN mkdir -p ${CODESERVER_EXTENSIONS_DIR}

RUN code-server --extensions-dir ${CODESERVER_EXTENSIONS_DIR} --install-extension utils/ms-python.python-2023.14.0.vsix
RUN code-server --extensions-dir ${CODESERVER_EXTENSIONS_DIR} --install-extension utils/ms-toolsai.jupyter-2023.3.100.vsix

RUN code-server --extensions-dir ${CODESERVER_EXTENSIONS_DIR} --install-extension utils/steoates.autoimport-1.5.3.vsix

Added thought:
since Container build happens with Internet access, we do not even need the extensions in the utils folder, I would assume. I'll check that out later. Doing an install at container build time with just the extension id, but nonetheless still with the --extensions-dir argument, would also work in this approach. Added bonus: no more manual finding out of compatibility of an extension.

Then, in the entrypoint script run-code-server.sh, a small modification giving bin/code-server the differing location of installed extensions:
Nothing else there, just that:

# Start server
start_process /usr/bin/code-server \
  --bind-addr 0.0.0.0:8787 \
  --disable-telemetry \
  --auth none \
  --disable-update-check \
  --extensions-dir /opt/app-root/share/code-server/extensions \
  /opt/app-root/src

working beautifully, I have multiple 10s of extensions availble baked-in. So no more liveness probe timeout issue on startup anymore as well.

Minor issue still with Jupyter notebook plugin, it wants to create a tmp folder in the extension's location, which now is of course not easily possible. Should be fixable ...

@atheo89 ah, settings.json is done this way, good

error 22:59:31.220: Failed to get baseUrl [Error: EACCES: permission denied, mkdir '/opt/app-root/share/code-server/extensions/ms-toolsai.jupyter-2023.3.100/tmp'] {
  errno: -13,
  code: 'EACCES',
  syscall: 'mkdir',
  path: '/opt/app-root/share/code-server/extensions/ms-toolsai.jupyter-2023.3.100/tmp'
}

solution: workspaceFolder (implicit?) being /opt/app-root/src, which is writable and the actual working with files workspace area ...

"jupyter.notebookFileRoot": "${workspaceFolder}"

Since an update quite some time ago (early 2023), ms-toolsai.jupyter can no longer properly launch a Jupyter kernel when Python and the notebook (Python) package are in scope because it tries to create a temp directory in its own extension directory

and probably also microsoft/vscode-jupyter#13525

Lastly, to summarize, your work in finding out the commands, your comments on versioning for extensions, how to find out which extension is compatible with a given code-server version (engine entry in package.json of an extension) really helped me get my airgapped image take off / working. Thank you all.

[jstourac]

Partially of topic - should we have a tracking Jira issue for this as it should be our primary tool?

[shalberd]

@harshad16

Would you like me to provide my findings to ODH Contrib Workbench in the form of a pull request and i.e. an additional snippet / folder / IDE bundle in contrib terms https://github.com/opendatahub-io-contrib/workbench-images/tree/main/snippets/ides for VS Code Airgapped?

I also updated pip to version 24.0, setuptools to version 69.1.0 and wheel to version 0.42.0 for base images. Location would be around here:

https://github.com/opendatahub-io-contrib/workbench-images/tree/main/base/c9s

In terms of the extensions that will be installed during container build time, I'll probably have a textfile with the base extensions you all use here, too, and another, optional text file for extensions the user can individually specifify by name. Language packs

coder/code-server#6168
coder/code-server#6688

are not working correctly when installed airgapped currently, but everything else, including Jupyter, a .venv needed so Jupyter extension uses the system python packages and other settings make my airgapped VS Code work flawlessly.

My thinking behind this is to first provide the working findings that in the end result in a custom podman ContainerFile and some minor modifications to the community, which could also yield feedback potentially.

[harshad16]

@shalberd this is amazing, thanks for checking on this 💯
i m sure your contribution to either this repo or upstream opendatahub-io-contrib would be appreciated.
we have this on in our repo, we can certainly implement this.

[atheo89]

This PR incorporate the changes that fix this issue

[shalberd]

nice, thank you