Animal Rescue ♥️😺 ♥️🐶 ♥️🐰 ♥️🐦 ♥️🐹

The sample repo for "Bullet-proof Microservices with Spring & Kubernetes" by Bella Bai and Oliver Hughes.

You can also checkout tag startdemo the following steps as we were going through in our talk.

Or you can check out the corresponding tag to get the code for different stages:

• startdemo: starting point
• basicauth: TLS + Ingress + Basic Auth
• oauth2-gateway: TLS + Ingress + OAuth2 with Spring Cloud Gateway
• oauth2-proxy: TLS + Ingress + OAuth2 with oauth2-proxy
• mTLS: mTLS

Please open an issue/PR if any of the tags or steps are inaccurate.

Have fun rescuing!

Requirements

• A Kubernetes cluster. Ideally running in a cloud environment (such as Google Kubernetes Engine or VMware TKGi) as the examples require internet facing DNS records.

• kubectl CLI

• helm CLI

  • Add the following repos following the steps:

    helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx  # for nginx ingress
    helm repo add jetstack https://charts.jetstack.io                       # for cert-manager
    helm repo add bitnami https://charts.bitnami.com/bitnami                # for external-dns
    helm repo add smallstep https://smallstep.github.io/helm-charts/        # for autocert

• skaffold CLI

• A top level domain name to use (our examples use the domain spring.animalrescue.online, you will need to change this to your own)

• You need to create an NS record for a subdomain of your top level domain. Our examples use an NS record pointing to Google Cloud DNS servers. You can use any DNS provider in this list

• Do a global find and replace in the k8s folder: s/spring.animalrescue.online/yourdomain.com/g

• Before the deployment, build the frontend artifact:

  ./gradlew :frontend:assemble

• GCP service account secret for using ExternalDNS. See this section You can manually create DNS entry and remove ExternalDNS from skaffold and kustomization

• OAuth2 application with GitHub or your favorite provider. See this section

• OAuth2 application with an OIDC provider. See this section

• The guide assume you have skaffold dev running to apply changes. You can also run skaffold run with each step.

Basic Auth on API

All the related sample code can be found on tag basicauth

Manual basic auth

1. Create secret

   kubectl create secret generic animal-rescue-basic --from-literal=username=alice  --from-literal=password=test

2. Use the secret in the container, use basic profile

   env:
     - name: SPRING_PROFILES_ACTIVE
       value: basic
     - name: ANIMAL_RESCUE_SECURITY_BASIC_PASSWORD
       valueFrom:
         secretKeyRef:
           name: animal-rescue-basic
           key: password
     - name: ANIMAL_RESCUE_SECURITY_BASIC_USERNAME
       valueFrom:
         secretKeyRef:
           name: animal-rescue-basic
           key: username

3. Deploy and verify basic auth working with the app, and partner app shows 401

4. Add basic auth configuration to external API deployment

             - name: ANIMAL_RESCUE_PASSWORD
               valueFrom:
                 secretKeyRef:
                   name: animal-rescue-basic
                   key: password
             - name: ANIMAL_RESCUE_USERNAME
               valueFrom:
                 secretKeyRef:
                   name: animal-rescue-basic
                   key: username

5. Edit server.js to use basic auth from secret

   const response = await axios.get(`${animalRescueBaseUrl}/api/animals`, {
       auth: {
           username: animalRescueUsername,
           password: animalRescuePassword
       }
   });

6. Partner API use the same secret to access /api/animals endpoint

7. Change the secret and rolling restart

Ingress + Basic Auth

1. Start fresh from the beginning.

2. Uncomment the ingress release in deploy/helm/releases section in skaffold.yaml

3. Create secret

   Generate auth file using the following command:

   mkdir k8s/ingress/secret
   cd k8s/ingress/secret
   htpasswd -c auth alice # Password is MD5 encrypted by default

   Create secret with secretGenerator in ingress kustomization (uncomment):

   secretGenerator:
   - name: ingress-basic-auth
     type: Opaque
     files:
     - secrets/auth
   
   generatorOptions:
     disableNameSuffixHash: true

4. Add basic auth annotation to animal-rescue-ingress yaml (uncomment)

     annotations:
       # type of authentication
       nginx.ingress.kubernetes.io/auth-type: basic
       # name of the secret that contains the user/password definitions
       nginx.ingress.kubernetes.io/auth-secret: ingress-basic-auth

5. Uncomment - k8s/ from deploy.kustomize.paths section in skaffold.yaml to include the ingress yaml

6. You need DNS records [spring.yourdomain | partner.spring.yourdomain | auth-external.yourdomain] and assign them to the ingress IP address. If you don't want to do it manually, you can use ExternalDNS to generate DNS entry.

Note for uninstall: you may need to fun the following command after helm uninstall

kubectl delete -A ValidatingWebhookConfiguration ingress-animal-rescue-ingress-nginx-admission

Note for running on GKE: I had to run the following command to enable webhook:

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin \
  --user $(gcloud config get-value account)

ExternalDNS

Helm install

Example helm values

1. Set up a hosted DNS with the provider of your choice. We use Google DNS. More info in this section
2. Pull down the service account json with gcloud cli :

   gcloud iam service-accounts keys create ./k8s/external-dns/secrets/gcp-dns-account-credentials.json --iam-account=$CLOUD_DNS_SA

3. Update helm values for your DNS provider. ExternalDNS has more information on integration with different providers in theirdoc
4. Uncomment resources[external-dns] in k8s/kustomization.yaml
5. Uncomment the ingress release in deploy/helm/releases section in skaffold.yaml
6. Add new ingress rule with new domain if you don't already have the DNS records for listed domains

Watch on DNS zone changes:

watch gcloud dns record-sets list --zone animal-rescue-zone 

Tail logs with stern:

stern external -n external-dns

The record will take a few minute to propogate. Keep pinging...

TLS

1. Uncomment the cert-manager release in deploy/helm/releases section in skaffold.yaml
2. Include letsencrypt-staging.yaml in ingress kustomization
3. Update ingress, uncomment code following the TODOs

   annotations:
    # In addition to existing auth annotations
    cert-manager.io/cluster-issuer: "letsencrypt"
    kubernetes.io/tls-acme: "true" # This annotation tells ingress to exclude that acme challenge path from authentication.
   
   # Add TLS enforcement
   spec:
    tls:
      - hosts:
          - fluffy.spring.animalrescue.online
        secretName: animal-rescue-certs
      - hosts:
          - partner.spring.animalrescue.online
        secretName: partner-certs

Check ingress gets certificate related events:

kubectl describe ingress animal-rescue-ingress

Check certificate status:

kubectl get certificates

Verify TLS:

curl http://fluffy.spring.animalrescue.online/api/animals # Should get `308 Permanent Redirect` back
curl https://fluffy.spring.animalrescue.online/api/animals -k # Should get `401` back
curl https://fluffy.spring.animalrescue.online/api/animals -k --user alice:test # Should get `200`response back

After verified that everything works fine, switch to use prod server.

• Add letsencrypt-prod.yaml to kustomization
• Use the prod issuer in ingress
• Update the secret names in the TLS section so cert manager can create new ones.

Visit the site again to see a trusted cert!

OAuth2

This talk was initially developed to use oauth2-proxy. However, we now suggest Spring Cloud Gateway as a better option because it's just another Spring Boot app.

Securing HTTP with Spring Cloud Gateway with any OIDC provider

All the related sample code can be found on tag oauth2-gateway

1. Create an OAuth2 App with Auth0
2. When creating the OAuth App with Auth0, make sure the Allowed Callback URLs include your own domain eg: https://gateway.spring.animalrescue.online/login/oauth2/code/auth0
3. Create the file oauth2-gateway/k8s/secrets/oauth2-credentials.txt, using the structure below:

client-id=${your-client-id}
client-secret=${your-client-secret}
issuer-uri=https://dev-nexus-demo.us.auth0.com/

1. Change the gateway ingress to use your domain: oauth2-gateway/k8s/gateway-ingress.yaml

skaffold run should deploy gateway, create ingress for it and enable TLS, creating DNS record with ExternalDNS. Then you can access the gateway url to find animal-rescue fully functional: https://gateway.spring.animalrescue.online

Securing HTTP with oauth2-proxy with GitHub

All the related sample code can be found on tag oauth2-proxy

1. Create an OAuth application in the GitHub website (or another OAuth provider such as google if you prefer but you will need to change the settings in k8s/oauth2-proxy/external-oauth2-proxy-helm-values.yaml) See the GitHub guide for setting up an OAuth application

2. When creating the OAuth App in GitHub, make sure the Authorization Callback URL points at your own domain eg: https://auth-external.spring.animalrescue.online/oauth2/callback

3. Create the directory k8s/oauth2-proxy/secrets (it is git ignored)

4. Create the file k8s/oauth2-proxy/secrets/oauth2-external-proxy-creds, using the structure below;

   cookie-secret=<<random value>>
   client-id=<<Your client ID from the GitHub OAuth application page>>
   client-secret=<<Your client secret from the GitHub OAuth application page>>

5. In the Helm values file k8s/oauth2-proxy/external-oauth2-proxy-helm-values.yaml change the email under restricted_access: to the email of your GitHub account

Setting up internal OAuth with an OIDC provider

The idea of this section is that you may want to protect internal resources using a different auth backend.

In our example we are using TKGi Kubernetes and its' internal UAA OAuth2 provider. You will need to modify the configuration in k8s/oauth2-proxy/internal-oauth2-proxy-helm-values.yaml and change oidc-issuer-url to point at your own service. The create an OAuth Client and add the client id and secret to the file k8s/oauth2-proxy/secrets/oauth2-internal-proxy-creds.

Install both oauth2-proxy and use them with ingress

1. Include oauth2-proxy in k8s/kustomization
2. Include oauth2 ingresses in ingress/kustomization
3. Uncomment the 2 oauth2-proxy releases in deploy/helm/releases section in skaffold.yaml
4. Open a browser to yourdomain.com and you should be presented with a GitHub authorization page and redirected to the animal-rescue site
5. Accessing any actuator endpoint yourdomain.com/actuator/* and you should be redirected to your choice of internal auth provider for authentication then redirected to the actuator endpoint.

mTLS with Autocert

Doc

1. Uncomment the autocert release in deploy/helm/releases section in skaffold.yaml

2. Label the namespace to enable autocert:

   kubectl label namespace animal-rescue autocert.step.sm=enabled

   Or Add the following metadata to namespace.yaml

   metadata:
     # Additional to existing metadata
     labels:
       autocert.step.sm: enabled

3. Annotate the node app

    spec:
      template:
        metadata:
          # Additional to existing metadata
          annotations:
            autocert.step.sm/name: partner-adoption-center.animal-rescue.svc.cluster.local

4. Check certs on container:

   set PARTNER_POD (kubectl get pods -l app=partner-adoption-center -o jsonpath='{$.items[0].metadata.name}')
   kubectl exec -it $PARTNER_POD -c partner-adoption-center -- ls /var/run/autocert.step.sm
   # Should see root.crt  site.crt  site.key
   kubectl exec -it $PARTNER_POD -c partner-adoption-center -- cat /var/run/autocert.step.sm/site.crt | step certificate inspect --short -
   # We can see the subject is what we set in the annotation, and the cert is valid for a day. Then the sidecar will take care of the renewal

5. Update the node app to be mTLS enabled. You can compare the file with the branch mtls to get the code that works, or follow this code example to come up with your own.

6. Update service port from 80 to 443

7. Deploy a CURL mTLS client:

   kubectl apply -f ./external-api/k8s/curl-mtls-client.yaml

   Or add it through kustomization.

8. Verify it work by checking the logs:

   stern curl-mtls-client

   Or do the curl manually:

   set MTLS_CLIENT (kubectl get pods -l app=curl-mtls-client -o jsonpath='{$.items[0].metadata.name}')
   
   k exec -it $MTLS_CLIENT -- curl -sS \
          --cacert /var/run/autocert.step.sm/root.crt \
          --cert /var/run/autocert.step.sm/site.crt \
          --key /var/run/autocert.step.sm/site.key \
          https://partner-adoption-center.animal-rescue.svc.cluster.local
   # Should return 200 with response
   
   kubectl exec -it $MTLS_CLIENT -- curl https://partner-adoption-center.animal-rescue.svc.cluster.local
   # Should fail on server cert validation
   
   kubectl exec -it $MTLS_CLIENT -- curl https://partner-adoption-center.animal-rescue.svc.cluster.local -k
   # Should fail on client cert validation

Google DNS setup

The kNative project has a good guide for setting up external DNS with Google Cloud that you can follow in order to get a K8s cluster up and running https://knative.dev/v0.15-docs/serving/using-external-dns-on-gcp/#set-up-kubernetes-engine-cluster-with-clouddns-readwrite-permissions