Add e2e test for train API

[helenxie-bit]

What this PR does / why we need it:
Add an e2e test in the test_e2e_train_api.py for the train API.

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes #

Checklist:

[coveralls]

Pull Request Test Coverage Report for Build 12449362407

Details

• 0 of 0 changed or added relevant lines in 0 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage remained the same at 100.0%

---

Totals
Change from base Build 12381609583:	0.0%
Covered Lines:	85
Relevant Lines:	85

---

💛 - Coveralls

[helenxie-bit]

Run the e2e test for the train API as a separate GitHub Action.

Can you try to separate them and see if issue will be resolved ? cc @kubeflow/wg-training-leads

@andreyvelich I've separated the e2e test for train API and now it works. Please review when you have time.

[andreyvelich]

Thank you for this overall lgtm, just small comment.
/assign @deepanker13 @kubeflow/wg-training-leads @Electronic-Waste

[deepanker13]

/lgtm
Thanks @helenxie-bit

[helenxie-bit]

I've updated the Kubernetes version to v1.31.4. Please review when you have time @andreyvelich @Electronic-Waste @kubeflow/wg-training-leads

[Electronic-Waste]

Basically LGTM. I left some comments for you @helenxie-bit

[Electronic-Waste]

Shall we change the Kubernetes version to be aligned with other ci tests? Like:

training-operator/.github/workflows/test-example-notebooks.yaml

Lines 16 to 18 in 69094e1

	matrix:
	kubernetes-version: ["v1.28.7", "v1.29.2", "v1.30.6"]
	python-version: ["3.9", "3.10", "3.11"]

[andreyvelich]

Just to save compute resources, I think for now we can just run this test on a single k8s version, since we run the rests E2E tests on the all versions.
WDYT @Electronic-Waste ?

[Electronic-Waste]

Yeah, I agree. Maybe we can select one k8s version from this list:)

[helenxie-bit]

OK, let me change the version to v1.30.6. And we can update it if needed in the future.

[andreyvelich]

given that we support 1.28-1.31, I would suggest that we run our integration tests on 1.29, 1.30, 1.31, we can update it in the following PR.
For the train API tests, I think running it on 1.31 should be sufficient.

[helenxie-bit]

Sounds good. So I think we will still keep the v1.31.4 version.

[andreyvelich]

Thank you for this effort @helenxie-bit!
/lgtm
/approve
/hold cancel

[google-oss-prow]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andreyvelich, tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [andreyvelich,tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment