Deploy KubeCF on Kind - Windows + WSL2 + Docker Addendum

[jimconner]

Not really an issue as such - more of a suggestion to improve the docs when using kind on Windows inside WSL2.

Ingress to docker containers works differently for Windows than it does on Linux so the default instructions which use the node IP as the system domain do not work correctly for people on Windows platform.

Here's the workaround that I used to get things up and working.

1. Create a cluster config for kind that sets up 80 & 443 as ingress ports.

cat << _EOF_ > cluster.yml
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  kubeadmConfigPatches:
  - |
    kind: InitConfiguration
    nodeRegistration:
      kubeletExtraArgs:
        node-labels: "ingress-ready=true"
  extraPortMappings:
  - protocol: TCP
    containerPort: 80
    hostPort: 80
  - protocol: TCP
    containerPort: 443
    hostPort: 443
containerdConfigPatches:
  - |-
    [plugins."io.containerd.grpc.v1.cri".containerd]
    disable_snapshot_annotations = true
_EOF_

2. Create the cluster specifying the cluster.yml config file.

kind create cluster --name kubecf --config=cluster.yml

3. After creating the values.yml file, edit it and change the system_domain to 127.0.0.1.nip.io

e.g. values.yml ends up looking something like this

system_domain: 127.0.0.1.nip.io
services:
  router:
    externalIPs:
    - 172.17.0.2
features:
  eirini:
    enabled: true
install_stacks: ["sle15"]

The rest of the install is as normal.
After install, cf api --skip-ssl-validation api.127.0.0.1.nip.io

From that point on, everything works properly. You can acess the apps you push to cf from your windows browser etc.

[jandubois]

If documentation for kind on Windows (and macOS) is added, it should also mention that it will only support Eirini. Diego requires XFS support for the grootfs, and the XFS kernel module is not installed in the Moby VM image that is used as the host in Docker for Desktop (on Windows and macOS), so containers cannot create XFS file systems.

[jimconner]

Thank you @jandubois - That nicely explains why I wasn't able to get diego working correctly.

[jandubois]

@jimconner At least on macOS you can use minikube with Virtualbox to run kubecf with Diego. I suspect it would work on Windows as well.

[mook-as]

On WSL2, it looks like the kernel (4.19.128-microsoft-standard) appears to have XFS support; at least, /proc/filesystems contains a xfs line. It also has btrfs, but I probably wouldn't trust it on a 4.x kernel…

Note that you'll need to configure Docker to use the WSL2 engine for this.

[jimconner]

@jandubois , @mook-as I've done some more digging into what is causing Diego to fail on my WSL2 setup (kernel is 4.19.128-microsoft-standard and XFS is enabled). In my particular case it seems to be the rep job that is failing (quite possibly because of my ingress setup with the sytem domain being 127.0.0.1).

{"timestamp":"2020-11-17T15:33:27.176695900Z","level":"error","source":"rep","message":"rep.failed-to-initialize-executor","data":{"error":"external networker up: exit status 1"}}

I thought I'd share my findings so far. I'm not planning on spending too much time investigating as it's the Eirini support that interests me most, but I'm happy to act as a tester if anyone thinks up a solution.