It's a collection of a few solution for Grafana - Kubernetes integration.
It's possible to access the Kubernetes API trough https://grafana.com/grafana/plugins/marcusolsson-json-datasource/, but access token to Kubernetes API must be set in Grafana, which is not too secure. Instead, a kubectl proxy can run in same namespace with read-only access, limited to a set of resources.
Grafana can present Loki logs, but if Loki (Promtail) is not deployed or Loki does not configured to collect logs from all Pods, Grafana can show Pod logs trough JSON API datasource, too. There is one trick: an nginx container transforms lines to JSON array by a simple script.
Table panel can have external link, which is configured on Pod listing panels to access an external Kubernetes dashboard.
The example setup looks like:
┌──────────────────────────────────────────┐ ┌──────────────────────┐
│ │ │ │
│ Grafana │ │ Adapter Pod │
│ │ │ │
│ ┌──────────────┐ ┌──────────────┐ │ │ ┌──────────────┐ │
│ │ │ │ │ │ │ │ │ │
│ │ Pod info: │ │ JSON API │ │ │ │ kubectl │ │
│ │ Table, Stat │◄───┤ datasurce │◄──┼─────┼───┤ ext │ │
┌────┼───┤ panel plugin │ │ plugin │ │ │ │ │ │
│ │ │ │ │ │ │ │ │ │ │
│ │ └──────────────┘ └──────────────┘ │ │ └──────▲───────┘ │
│ │ │ │ │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │ ┌──────┴───────┐ │ ┌──────────────┐
│ │ │ │ │ │ │ │ │ │ │ │ │
│ │ │ Pod info: │ │ JSON API │ │ │ │ kubectl │ │ │ Kubernetes │
│ │ │ Table, Stat │◄───┤ datasurce │◄──┼─────┼───┤ proxy │◄──┼─────┤ API │
├────┼───┤ panel plugin │ │ plugin │ │ │ │ │ │ │ │
│ │ │ │ ┌─┤ │ │ │ │ │ │ │ │
│ │ └──────────────┘ │ └──────────────┘ │ │ └──────────────┘ │ └──────┬───────┘
│ │ │ │ │ │ │
│ │ ┌──────────────┐ │ ┌──────────────┐ │ │ ┌──────────────┐ │ │
│ │ │ │◄─┘ │ │ │ │ │ │ │ │
│ │ │ Pod logs: │ │ JSON API │ │ │ │ nginx, │ │ │
│ │ │ Table │◄───┤ datasurce │◄──┼─────┼───┤ lines to │◄──┼────────────┘
│ │ │ panel plugin │ │ plugin │ │ │ │ JSON │ │
│ │ │ │ │ │ │ │ │ │ │
│ │ └──────────────┘ └──────────────┘ │ │ └──────────────┘ │
│ │ ▲ │ │ │
│ │ │ │ └──────────────────────┘
│ │ │ │
│ │ │ │
│ │ │ │
│ │ │ │ ┌──────────────────────┐
│ │ │ │ │ │
│ │ │ │ │ Loki Pod │
│ │ │ │ │ │
│ │ │ ┌──────────────┐ │ │ ┌──────────────┐ │
│ │ │ │ │ │ │ │ │ │
│ │ │ │ Loki │ │ │ │ │ │
│ │ └───────────┤ datasurce │◄──┼─────┼───┤ Loki │ │
│ │ │ plugin │ │ │ │ │ │
│ │ │ │ │ │ │ │ │
│ │ └──────────────┘ │ │ └──────────────┘ │
│ │ │ │ │
│ └──────────────────────────────────────────┘ └──────────────────────┘
│
│
│ ┌──────────────┐
│ │ │
│ link to │ Kiali │
└───────────────►│ or other │
│ dashboard │
│ │
└──────────────┘
There are several options to run a demo Kubernetes on laptop, see more details here: Environment for comparing several on-premise Kubernetes distributions (K3s, MicroK8s, KinD, kubeadm). The make all command deploys JSON API datasource plugin with configuration. The pgillich-tree-panel plugin is also included.
Deployment files of Adapter Pod can be found in kubernetes/monitoring directory. These files can be deployed by kubectl apply -k kubernetes/monitoring command.
The kubect proxy is configured to deny any modification requests, see at kubectl-proxy-deployment.yaml. Other access limitations are configured in kubectl-proxy-clusterRole.yaml.
A simple LUA code transforms the raw log lines to JSON array, see kubectl-proxy-openresty-config.yaml.
Example requests to the adapter pod from the Grafana container:
export GRAFANA_POD=$(kubectl get pod -n monitoring -l 'app.kubernetes.io/name=grafana' -o name | sed 's#^pod/##g');
kubectl exec -n monitoring ${GRAFANA_POD} -c grafana -- /bin/sh -c 'wget -q -O - http://kubectl-proxy/api/v1/namespaces/monitoring/pods'
kubectl exec -n monitoring ${GRAFANA_POD} -c grafana -- /bin/sh -c 'wget -q -O - http://kubectl-proxy:8002/api/v1/namespaces/monitoring/pods/alertmanager-prometheus-st
ack-kube-prom-alertmanager-0/log?container=alertmanager'
kubectl exec -n monitoring ${GRAFANA_POD} -c grafana -- /bin/sh -c 'wget -q -O - http://kubectl-proxy:8003/api/v1/namespaces/monitoring/pods'Example configurations:
Raw Kubernetes responses should be transformed to a flat JSON array, substituting the missing values. Fortunately, JSON API datasource supports JSONata. Example JSONata expression for transformation and substitution:
$map(items, function($v) {{"namespace": $v.metadata.namespace, "name": $v.metadata.name, "appName": $v.metadata.labels."app.kubernetes.io/name" ? $v.metadata.labels."app.kubernetes.io/name" : "-", "statusPhase": $v.status.phase, "containerCount": $count($v.spec.containers), "containerImage": $join($v.spec.containers[*].image, " "), "containerState": $v.status.containerStatuses ? $string($v.status.containerStatuses[*].state) : "-"}})
JSONAta expressions can be evaluated by jfq. It can be installed by below command:
npm install --global jfqExample usage:
jfq '$filter($each(items{status.phase: $count(status.phase)}, function($v, $k) {{"status": $k, "count": $v}}), function($v) {$v.status="Running" or $v.status="Succeeded"})' /tmp/pods.jsonImportable Grafana Dashboard files can be found in grafana/dashboards directory. The *(ext).json dashboards use kubectl-ext container as a data source.
Pod table panels are configured with external links, which uses the external server URL from a Dashboard variable. Example screenshot with link to Kiali Dashboard:
Flux does not have maintained UI, but Grafana can be used to show the current state, for example:
Example sceenshot:
