RHOAIENG-16076: tests(gha): run Makefile tests on opendatahub-io/notebooks Github Actions #775

jiridanek · 2024-11-22T17:07:46Z

https://issues.redhat.com/browse/RHOAIENG-16076

This is building up on the preparatory work merged in

RHOAIENG-16076: tests(gha): use the same registry, tag, and output image name in all workflow tasks #780
RHOAIENG-16076: tests(gha): pre-pull trivy vulnerabilities db to prevent failures to download later #777
RHOAIENG-16076: tests(gha): change to using rootful podman, so that we can share containers/storage with cri-o later #782
RHOAIENG-16076: tests(gha): install a cri-o backed kubernetes for running Makefile tests #783

Description

Deploys cri-o backed kubernetes with kubeadm to run our Makefile tests.

I considered using KinD, or Microshift, or Minikube, or OpenShift Local, or some other Kubernetes, but decided not to, because

Microshift still requires Red Hat pull secret, so it's hard to run this for PRs
- There is something new there, though https://github.com/openshift/microshift/tree/main/okd/src
OpenShift Local takes long time to start, the OpenShift version needs pull-secret, and the OKD version is not getting much love
KinD, etc. with CRI-O requires custom node image https://github.com/cri-o/cri-o/blob/release-1.31/tutorials/crio-in-kind.md#cri-o-in-kind

The reason why I need cri-o is that I want to share the podman-built images into the cluster without having to copy them. Copying images is difficult because the images can be huge.

seems that k3d would've been also possible, https://devopstales.github.io/kubernetes/k3s-crio/

Known issues that I do intend to fix

Resolving raw.githubusercontent.com (raw.githubusercontent.com)... failed: Name or service not known.
wget: unable to resolve host address ‘raw.githubusercontent.com’

This needs a retry, because it happens from time to time. Also, we should not fetch test data from the main branch of the repo, we should use what's in the current checkout.

This went away after I redid my Kubernetes network. I had this problems with flannel, and since then I switched to a plain bridge with firewall masquerade. I think that (when I noticed this problem the first time, or maybe at some later point) my pods were completely unable to connect to the outside (also unable to connect to other pods, like the coredns pod, but I was dealing with other issues so I mistakenly thought the networking issues are a flakiness.

WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection object at 0x7f95ec277fd0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /simple/papermill/
ERROR: Could not find a version that satisfies the requirement papermill (from versions: none)
Notice:  A new release of pip is available: 24.2 -> 24.3.1
Notice:  To update, run: pip install --upgrade pip
ERROR: No matching distribution found for papermill
command terminated with exit code 1

Could it be that freshly started pod has spotty network and I need to wait?

Looks like the same problem as above. My pods were completely offline. The bridge solution needs no waiting!

Known issues (that I don't intend to fix for this PR)

GHA for a pull request skips intermediate images that have their own tests. Those tests will not be run, only leaf images in the dependency chain are having their tests run. (The push GHA tests everything.)

How Has This Been Tested?

https://github.com/jiridanek/notebooks/actions/runs/12064274658

Merge criteria:

The commits are squashed in a cohesive manner and have meaningful messages.
Testing instructions have been added in the PR body (for PRs involving changes that are not immediately obvious).
The developer has manually tested the changes and verified that the changes work

base/ubi9-python-3.11/Dockerfile

…761 (comment) when merging the work

… let's do that

jiridanek · 2024-11-28T07:00:25Z

.github/workflows/build-notebooks-TEMPLATE.yaml

@@ -46,6 +46,8 @@ jobs:
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

+      # region Free up disk space


This is the VS Code-compatible syntax for collapsable regions. Allows to collapse parts of the file in IntelliJ and VS Code

…ations

…stry) and the symbolic links apparently needed to deploy rocm stuff

jiridanek · 2024-11-28T10:53:59Z

/test ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror
/test ci/prow/rocm-notebooks-e2e-tests

jiridanek · 2024-11-28T11:21:55Z

/test ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror

jiridanek · 2024-11-28T11:22:27Z

/test rocm-notebooks-e2e-tests

jiridanek · 2024-11-28T11:23:39Z

/test pull-ci-opendatahub-io-notebooks-main-notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror

jiridanek · 2024-11-28T13:16:59Z

/test rocm-notebooks-e2e-tests

atheo89 · 2024-11-28T14:30:15Z

Great work Jiri!
/lgtm

openshift-ci · 2024-11-28T15:25:06Z

@jiridanek: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/runtimes-ubi9-e2e-tests	`e85aa9e`	link	true	`/test runtimes-ubi9-e2e-tests`
ci/prow/notebooks-ubi9-e2e-tests	`e85aa9e`	link	true	`/test notebooks-ubi9-e2e-tests`
ci/prow/codeserver-notebook-e2e-tests	`e85aa9e`	link	true	`/test codeserver-notebook-e2e-tests`
ci/prow/notebook-cuda-jupyter-tf-ubi9-python-3-11-pr-image-mirror	`e85aa9e`	link	true	`/test notebook-cuda-jupyter-tf-ubi9-python-3-11-pr-image-mirror`
ci/prow/notebook-jupyter-pytorch-ubi9-python-3-11-pr-image-mirror	`e85aa9e`	link	true	`/test notebook-jupyter-pytorch-ubi9-python-3-11-pr-image-mirror`
ci/prow/runtime-intel-tf-ubi9-python-3-11-pr-image-mirror	`e85aa9e`	link	true	`/test runtime-intel-tf-ubi9-python-3-11-pr-image-mirror`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

jiridanek · 2024-11-28T16:04:13Z

/approve

openshift-ci · 2024-11-28T16:04:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jiridanek

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [jiridanek]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jiridanek · 2024-11-28T16:05:07Z

/override ci/prow/images

openshift-ci · 2024-11-28T16:05:13Z

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/images

In response to this:

/override ci/prow/images

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jiridanek · 2024-11-28T16:14:06Z

/override build (rocm-jupyter-pytorch-ubi9-python-3.9) / build

jiridanek · 2024-11-28T16:14:12Z

/override "build (rocm-jupyter-pytorch-ubi9-python-3.9) / build"

openshift-ci · 2024-11-28T16:14:18Z

@jiridanek: Overrode contexts on behalf of jiridanek: build (rocm-jupyter-pytorch-ubi9-python-3.9) / build

In response to this:

/override "build (rocm-jupyter-pytorch-ubi9-python-3.9) / build"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jiridanek · 2024-11-28T16:15:05Z

/override "build (rocm-jupyter-tensorflow-ubi9-python-3.11) / build"

openshift-ci · 2024-11-28T16:15:11Z

@jiridanek: Overrode contexts on behalf of jiridanek: build (rocm-jupyter-tensorflow-ubi9-python-3.11) / build

In response to this:

/override "build (rocm-jupyter-tensorflow-ubi9-python-3.11) / build"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jiridanek · 2024-11-28T16:21:35Z

/override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror

openshift-ci · 2024-11-28T16:21:42Z

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror

In response to this:

/override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-11-pr-image-mirror

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jiridanek · 2024-11-28T16:22:17Z

/override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror
/override ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror
/override ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-9-pr-image-mirror

openshift-ci · 2024-11-28T16:22:26Z

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror, ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror, ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-9-pr-image-mirror

In response to this:

/override ci/prow/notebook-rocm-jupyter-pyt-ubi9-python-3-9-pr-image-mirror
/override ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-11-pr-image-mirror
/override ci/prow/notebook-rocm-jupyter-tf-ubi9-python-3-9-pr-image-mirror

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jiridanek · 2024-11-28T16:22:46Z

/override ci/prow/rocm-notebooks-e2e-tests

openshift-ci · 2024-11-28T16:22:51Z

@jiridanek: Overrode contexts on behalf of jiridanek: ci/prow/rocm-notebooks-e2e-tests

In response to this:

/override ci/prow/rocm-notebooks-e2e-tests

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jstourac · 2024-11-28T17:15:14Z

jupyter/rocm/pytorch/ubi9-python-3.11/kustomize/base/kustomization.yaml

 commonLabels:
-  app: rocm-jupyter-pytorch-ubi9-python-3-11
+  app: jupyter-rocm-pytorch-ubi9-python-3-11


Hey, didn't check - what does this affects? Is it safe? 🤔

does not run without this change in gha, and I did not manage to make the tests run in openshift-ci, so, yeah, I think it makes things better overall

…ebooks Github Actions (opendatahub-io#775) * RHOAIENG-16076: tests(gha): run Makefile tests in GitHub Actions * fixup, looks like I lost the second changed line from opendatahub-io#761 (comment) when merging the work * fixup, linter wants space in the comments; IntelliJ is ok with it, so let's do that * fixup, add reference to OpenShift CI for the source of the make invocations * fixup, the ifNotPresent pull policy (for PR checks without image registry) and the symbolic links apparently needed to deploy rocm stuff (cherry picked from commit b3d8af0)

…books Github Actions (opendatahub-io#775) * RHOAIENG-16076: tests(gha): run Makefile tests in GitHub Actions * fixup, looks like I lost the second changed line from opendatahub-io#761 (comment) when merging the work * fixup, linter wants space in the comments; IntelliJ is ok with it, so let's do that * fixup, add reference to OpenShift CI for the source of the make invocations * fixup, the ifNotPresent pull policy (for PR checks without image registry) and the symbolic links apparently needed to deploy rocm stuff (cherry picked from commit b3d8af0)

jiridanek added the tide/merge-method-squash Denotes a PR that should be squashed by tide when it merges. label Nov 22, 2024

openshift-ci bot requested review from caponetto and jstourac November 22, 2024 17:08

jiridanek commented Nov 22, 2024

View reviewed changes

base/ubi9-python-3.11/Dockerfile Outdated Show resolved Hide resolved

jiridanek added the trivy-scan This label that allows trivy to create a security report on the pull requests label Nov 22, 2024

jiridanek marked this pull request as draft November 24, 2024 17:33

openshift-ci bot added the do-not-merge/work-in-progress label Nov 24, 2024

jiridanek mentioned this pull request Nov 26, 2024

RHOAIENG-16076: tests(gha): install a cri-o backed kubernetes for running Makefile tests #783

Merged

3 tasks

openshift-merge-robot added the needs-rebase label Nov 27, 2024

RHOAIENG-16076: tests(gha): run Makefile tests in GitHub Actions

f4ff6c7

jiridanek force-pushed the jd_rebased_run_tests_gha branch from e85aa9e to f4ff6c7 Compare November 28, 2024 06:38

openshift-merge-robot removed the needs-rebase label Nov 28, 2024

jiridanek added 2 commits November 28, 2024 07:54

fixup, looks like I lost the second changed line from opendatahub-io#…

d8bc7f3

…761 (comment) when merging the work

fixup, linter wants space in the comments; IntelliJ is ok with it, so…

3e2328e

… let's do that

jiridanek commented Nov 28, 2024

View reviewed changes

fixup, add reference to OpenShift CI for the source of the make invoc…

90ffad7

…ations

jiridanek removed the do-not-merge/work-in-progress label Nov 28, 2024

jiridanek marked this pull request as ready for review November 28, 2024 07:04

fixup, the ifNotPresent pull policy (for PR checks without image regi…

0040db8

…stry) and the symbolic links apparently needed to deploy rocm stuff

This comment was marked as outdated.

Sign in to view

This comment was marked as off-topic.

Sign in to view

openshift-ci bot assigned atheo89 Nov 28, 2024

openshift-ci bot added the lgtm label Nov 28, 2024

openshift-ci bot added the approved label Nov 28, 2024

This comment was marked as outdated.

Sign in to view

openshift-merge-bot bot merged commit b3d8af0 into opendatahub-io:main Nov 28, 2024
15 of 17 checks passed

jiridanek deleted the jd_rebased_run_tests_gha branch November 28, 2024 16:29

jstourac reviewed Nov 28, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RHOAIENG-16076: tests(gha): run Makefile tests on opendatahub-io/notebooks Github Actions #775

RHOAIENG-16076: tests(gha): run Makefile tests on opendatahub-io/notebooks Github Actions #775

jiridanek commented Nov 22, 2024 •

edited

Loading

jiridanek Nov 28, 2024

jiridanek commented Nov 28, 2024

This comment was marked as outdated.

jiridanek commented Nov 28, 2024

This comment was marked as outdated.

This comment was marked as off-topic.

jiridanek commented Nov 28, 2024

jiridanek commented Nov 28, 2024

jiridanek commented Nov 28, 2024

atheo89 commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024 •

edited

Loading

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

This comment was marked as outdated.

jiridanek commented Nov 28, 2024

This comment was marked as outdated.

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

This comment was marked as outdated.

This comment was marked as outdated.

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

jstourac Nov 28, 2024

jiridanek Nov 28, 2024

RHOAIENG-16076: tests(gha): run Makefile tests on opendatahub-io/notebooks Github Actions #775

RHOAIENG-16076: tests(gha): run Makefile tests on opendatahub-io/notebooks Github Actions #775

Conversation

jiridanek commented Nov 22, 2024 • edited Loading

Description

Known issues that I do intend to fix

Known issues (that I don't intend to fix for this PR)

How Has This Been Tested?

Merge criteria:

jiridanek Nov 28, 2024

Choose a reason for hiding this comment

jiridanek commented Nov 28, 2024

This comment was marked as outdated.

jiridanek commented Nov 28, 2024

This comment was marked as outdated.

This comment was marked as off-topic.

jiridanek commented Nov 28, 2024

jiridanek commented Nov 28, 2024

jiridanek commented Nov 28, 2024

atheo89 commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024 • edited Loading

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

This comment was marked as outdated.

jiridanek commented Nov 28, 2024

This comment was marked as outdated.

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

This comment was marked as outdated.

This comment was marked as outdated.

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

jiridanek commented Nov 28, 2024

openshift-ci bot commented Nov 28, 2024

jstourac Nov 28, 2024

Choose a reason for hiding this comment

jiridanek Nov 28, 2024

Choose a reason for hiding this comment

jiridanek commented Nov 22, 2024 •

edited

Loading

openshift-ci bot commented Nov 28, 2024 •

edited

Loading