KubeVirt on Nova

This is a demo of scheduling KubeVirt across a fleet of 2 Kubernetes clusters using Nova.

KubeVirt-Nova-demo.mp4

Prereq: Deploy Nova on KIND

Deploy Nova on Kind using try-nova repo.

Grab KubeConfig for {Nova, Workload Cluster 1, Workload Cluster 2} from Try-Nova deployment.

export TRY_NOVA_REPO_ROOT=$HOME/github.com/elotl/try-nova

cp $TRY_NOVA_REPO_ROOT/kubeconfig-cp .
cp $TRY_NOVA_REPO_ROOT/kubeconfig-workload-1 .
cp $TRY_NOVA_REPO_ROOT/kubeconfig-workload-2 .

Update KUBECONFIG to include Nova and workload clusters.

export KUBECONFIG=$HOME/.nova/nova/nova-kubeconfig:$PWD/kubeconfig-cp:$PWD/kubeconfig-workload-1:$PWD/kubeconfig-workload-2

Make sure your workload clusters are healthy.

kubectl --context=nova get clusters

You should see two workload clusters in Ready state set to True.

NAME    K8S-VERSION   K8S-CLUSTER   REGION   ZONE   READY   IDLE    STANDBY
wlc-1   1.30          wlc-1                         True    False   False
wlc-2   1.30          wlc-2                         True    False   False

Create new CRDs created by KubeVirt in Nova

kubectl --context=nova create -f ./crd-definitions/

Spread/Duplicate KubeVirt Namespace on all workload clusters

Create Spread/Duplicate Schedule Policy for kubevirt Namespace in Nova.

kubectl --context=nova apply -f ./policies/namespace-policy.yaml
kubectl --context=nova apply -f ./deploy-scripts/namespace.yaml

Verify kubevirt namespace has been spread to all workload clusters

KUBECONFIG=./kubeconfig-workload-1 kubectl get ns
KUBECONFIG=./kubeconfig-workload-2 kubectl get ns

You should see kubevirt namespace in both the workload clusters.

Spread/Duplicate KubeVirt Operator on all workload clusters

Create Spread/Duplicate policy for KubeVirt

kubectl --context=nova apply -f ./policies/kubevirt-policy.yaml

Get latest KubeVirt version

export KUBEVIRT_VERSION=$(curl -s https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)
echo "Deploying KubeVirt version " $KUBEVIRT_VERSION

Spread/Duplicate KubeVirt Operator on all workload clusters

kubectl --context=nova apply -f "https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/kubevirt-operator.yaml" 
kubectl --context=nova label crd kubevirts.kubevirt.io app.kubernetes.io/component=kubevirt
kubectl --context=nova label deployment virt-operator app.kubernetes.io/component=kubevirt -n kubevirt
kubectl --context=nova label priorityclass kubevirt-cluster-critical app.kubernetes.io/component=kubevirt -n kubevirt
kubectl --context=nova label clusterrole kubevirt-operator app.kubernetes.io/component=kubevirt
kubectl --context=nova label clusterrolebinding kubevirt-operator app.kubernetes.io/component=kubevirt
kubectl --context=nova label serviceaccount kubevirt-operator app.kubernetes.io/component=kubevirt -n kubevirt
kubectl --context=nova label role kubevirt-operator app.kubernetes.io/component=kubevirt -n kubevirt
kubectl --context=nova label rolebinding kubevirt-operator-rolebinding app.kubernetes.io/component=kubevirt -n kubevirt

Spread/Duplicate KubeVirt CR on all workload clusters

kubectl --context=nova apply -f "https://github.com/kubevirt/kubevirt/releases/download/${KUBEVIRT_VERSION}/kubevirt-cr.yaml"
kubectl --context=nova label kubevirt.kubevirt.io  kubevirt app.kubernetes.io/component=kubevirt -n kubevirt

Wait for 5 minutes for KubeVirt components to come up

KubeVirt should be in "Deployed" state on workload clusters

kubectl --context=nova get kubevirt kubevirt -n kubevirt
KUBECONFIG=kubeconfig-workload-1 kubectl get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.phase}"
KUBECONFIG=kubeconfig-workload-2 kubectl get kubevirt.kubevirt.io/kubevirt -n kubevirt -o=jsonpath="{.status.phase}"

Schedule a test VM to workload cluster "wlc-1" to test our setup

kubectl --context=nova apply -f ./policies/testvm-policy.yaml
kubectl --context=nova apply -f "https://kubevirt.io/labs/manifests/vm.yaml"
kubectl --context=nova label virtualmachine testvm app.kubernetes.io/component=testvm
kubectl --context=nova get virtualmachine testvm
KUBECONFIG=kubeconfig-workload-1 kubectl get virtualmachine
KUBECONFIG=kubeconfig-workload-2 kubectl get virtualmachine

Start/Stop testvm via Nova

kubectl --context=nova patch virtualmachine testvm --type merge -p '{"spec":{"runStrategy": "Always"}}'
kubectl --context=nova get virtualmachine testvm
KUBECONFIG=kubeconfig-workload-1 kubectl get virtualmachine

kubectl --context=nova patch virtualmachine testvm --type merge -p '{"spec":{"runStrategy": "Halted"}}'
kubectl --context=nova get virtualmachine testvm
KUBECONFIG=kubeconfig-workload-1 kubectl get virtualmachine

Teardown

Delete kubevirt namespace.

kubectl --context=nova delete -f ./deploy-scripts/namespace.yaml

This should delete kubevirt namespace on the workload clusters.

Optionally, teardown Kind clusters.

$TRY_NOVA_REPO_ROOT/scripts/teardown_kind.sh