Demo OpenShift autoscaling with gpu machine sets

Show how to autoscale GPU resources on OpenShift in AWS using the following...

Objects:

• MachineSet
• MachineSetAutoScaler
• ClusterAutoscaler

Operators:

• Node Feature Discovery Operator (NFD)
• NVIDIA GPU Operator

Prerequisites - Get a cluster

• OpenShift 4.16+
  • role: cluster-admin

Red Hat Demo Platform - Options (Tested) Below

NOTE: The node sizes below are the recommended minimum to select for provisioning

• AWS with OpenShift Open Environment
  • 1 x Control Plane - m6a.2xlarge
  • 0 x Workers - m6a.2xlarge
• One Node OpenShift
  • 1 x Control Plane - m6a.2xlarge
• OpenShift GitOps Blank Environment
  • 1 x Control Plane - 16 cores, 64Gi

Getting Started

Install the OpenShift Web Terminal

The following icon should appear in the top right of the OpenShift web console after you have installed the operator. Clicking this icon launches the web terminal.

NOTE: Reload the page in your browser if you do not see the icon after installing the operator.

Make the enhanced web terminal permanent

# apply the enhanced web terminal
oc apply -k https://github.com/redhat-na-ssa/demo-ocp-autoscale-gpu/demo/web-terminal

# delete old web terminal
$(wtoctl | grep 'oc delete')

NOTE: The following commands are available in an enhanced web terminal.

Quickstart

Run directly from remote

until oc apply -k https://github.com/redhat-na-ssa/demo-ocp-autoscale-gpu/demo; do : ; done

Query the status of the cluster autoscaler

oc -n openshift-machine-api \
  describe cm cluster-autoscaler-status

Uninstall

oc delete -k demo/uninstall

Reset the web terminal

oc apply -k components/operators/web-terminal/reset-default

Known Issues

NOTE: If you have manually installed operators through the OpenShift web console before running this automation there may be duplicate operator groups.

Manually Remove any duplicate operator groups ending in -xxxxx or use the following command in the enhanced web terminal.

ocp_fix_duplicate_operator_groups

Manual Steps

Manual steps for configuring a machineset for GPUs

Links

• OpenShift Docs - Autoscaling