fix(helm): Various fixes and enhancements for bloom components #14128

chaudum · 2024-09-12T11:44:57Z

What this PR does / why we need it:

Fix: Do not create bloom Deployment/Statefulset if replicas=0 (fixes helm: bloom-builder deployment created always #14090)
Fix: Configure PV claims for bloom gateway and bloom builder correctly (fixes bloom-builder: new component cannot start up because helm chart is not creating /var/loki path #14082)
Fix: Configure emphemeral storage for bloom builder working directory (fixes bloom-builder: panic: mkdir /tmp/bloom-block-1802389595: read-only file system #14084)
Add required configuration for bloom builder and bloom gateway

Checklist

Reviewed the CONTRIBUTING.md guide (required)
Documentation added
Tests updated
Title matches the required conventional commits format, see here
- Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
For Helm chart changes bump the Helm chart version in production/helm/loki/Chart.yaml and update production/helm/loki/CHANGELOG.md and production/helm/loki/README.md. Example PR
If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

Signed-off-by: Christian Haudum <[email protected]>

diranged · 2024-09-12T15:40:38Z

production/helm/loki/templates/bloom-builder/service-bloom-builder-headless.yaml

@@ -1,5 +1,5 @@
 {{- $isDistributed := eq (include "loki.deployment.isDistributed" .) "true" -}}
-{{- if $isDistributed -}}
+{{- if (and $isDistributed (gt (int .Values.bloomBuilder.replicas) 0)) -}}


I think you need an or .Values.bloomBuilder.autoscaling.enabled as well ... because we explicitly leave replicas: 0 and set autosacling.enabled in our environment

Good catch! Haven't thought about that.

diranged

Mostly good, one comment.. thanks for addressing these issues!

Signed-off-by: Christian Haudum <[email protected]>

github-actions · 2024-09-17T14:52:31Z

Kubernetes Manifest Diff Summary

Scenario: default-single-binary-values (Added: 0, Modified: 11, Removed: 3)

Summary:

Added: 0
Modified: 11
Removed: 3

Added Files

No added files

Modified Files

loki/templates/query-scheduler/deployment-query-scheduler.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/query-scheduler/deployment-query-scheduler.yaml	2024-09-17 14:52:24.366725800 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/query-scheduler/deployment-query-scheduler.yaml	2024-09-17 14:52:26.466752946 +0000
***************
*** 27,33 ****
 template:
 metadata:
 annotations:
! checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 27,33 ----
 template:
 metadata:
 annotations:
! checksum/config: 46367dc8cfea5f01466a79c988674cf14664a9c6d78c708f52b9cf30245b69b8
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/querier/deployment-querier.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/querier/deployment-querier.yaml	2024-09-17 14:52:24.366725800 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/querier/deployment-querier.yaml	2024-09-17 14:52:26.466752946 +0000
***************
*** 28,34 ****
 template:
 metadata:
 annotations:
! checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 28,34 ----
 template:
 metadata:
 annotations:
! checksum/config: 46367dc8cfea5f01466a79c988674cf14664a9c6d78c708f52b9cf30245b69b8
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/ingester/statefulset-ingester-zone-c.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-c.yaml	2024-09-17 14:52:24.366725800 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-c.yaml	2024-09-17 14:52:26.466752946 +0000
***************
*** 34,40 ****
 template:
 metadata:
 annotations:
! checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 34,40 ----
 template:
 metadata:
 annotations:
! checksum/config: 46367dc8cfea5f01466a79c988674cf14664a9c6d78c708f52b9cf30245b69b8
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/ingester/statefulset-ingester-zone-a.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-a.yaml	2024-09-17 14:52:24.366725800 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-a.yaml	2024-09-17 14:52:26.466752946 +0000
***************
*** 34,40 ****
 template:
 metadata:
 annotations:
! checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 34,40 ----
 template:
 metadata:
 annotations:
! checksum/config: 46367dc8cfea5f01466a79c988674cf14664a9c6d78c708f52b9cf30245b69b8
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/ingester/statefulset-ingester-zone-b.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-b.yaml	2024-09-17 14:52:24.366725800 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/ingester/statefulset-ingester-zone-b.yaml	2024-09-17 14:52:26.466752946 +0000
***************
*** 34,40 ****
 template:
 metadata:
 annotations:
! checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 34,40 ----
 template:
 metadata:
 annotations:
! checksum/config: 46367dc8cfea5f01466a79c988674cf14664a9c6d78c708f52b9cf30245b69b8
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/config.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/config.yaml	2024-09-17 14:52:24.362725748 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/config.yaml	2024-09-17 14:52:26.462752895 +0000
***************
*** 15,20 ****
--- 15,28 ----
 config.yaml: |
 
 auth_enabled: true
+ bloom_build:
+ builder:
+ planner_address: loki-test-chart-name-bloom-planner-headless.default.svc.cluster.local:9095
+ enabled: false
+ bloom_gateway:
+ client:
+ addresses: dnssrvnoa+_grpc._tcp.loki-test-chart-name-bloom-gateway-headless.default.svc.cluster.local
+ enabled: false
 chunk_store_config:
 chunk_cache_config:
 background:
***************
*** 102,107 ****
--- 110,117 ----
 http_server_read_timeout: 600s
 http_server_write_timeout: 600s
 storage_config:
+ bloom_shipper:
+ working_directory: /var/loki/data/bloomshipper
 boltdb_shipper:
 index_gateway_client:
 server_address: dns+loki-test-chart-name-index-gateway-headless.default.svc.cluster.local:9095

loki/templates/query-frontend/deployment-query-frontend.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/query-frontend/deployment-query-frontend.yaml	2024-09-17 14:52:24.366725800 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/query-frontend/deployment-query-frontend.yaml	2024-09-17 14:52:26.466752946 +0000
***************
*** 27,33 ****
 template:
 metadata:
 annotations:
! checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 27,33 ----
 template:
 metadata:
 annotations:
! checksum/config: 46367dc8cfea5f01466a79c988674cf14664a9c6d78c708f52b9cf30245b69b8
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/distributor/deployment-distributor.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/distributor/deployment-distributor.yaml	2024-09-17 14:52:24.366725800 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/distributor/deployment-distributor.yaml	2024-09-17 14:52:26.466752946 +0000
***************
*** 28,34 ****
 template:
 metadata:
 annotations:
! checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 28,34 ----
 template:
 metadata:
 annotations:
! checksum/config: 46367dc8cfea5f01466a79c988674cf14664a9c6d78c708f52b9cf30245b69b8
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/index-gateway/statefulset-index-gateway.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/index-gateway/statefulset-index-gateway.yaml	2024-09-17 14:52:24.366725800 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/index-gateway/statefulset-index-gateway.yaml	2024-09-17 14:52:26.466752946 +0000
***************
*** 26,32 ****
 template:
 metadata:
 annotations:
! checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 26,32 ----
 template:
 metadata:
 annotations:
! checksum/config: 46367dc8cfea5f01466a79c988674cf14664a9c6d78c708f52b9cf30245b69b8
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/compactor/statefulset-compactor.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/compactor/statefulset-compactor.yaml	2024-09-17 14:52:24.366725800 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/compactor/statefulset-compactor.yaml	2024-09-17 14:52:26.466752946 +0000
***************
*** 29,35 ****
 template:
 metadata:
 annotations:
! checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 29,35 ----
 template:
 metadata:
 annotations:
! checksum/config: 46367dc8cfea5f01466a79c988674cf14664a9c6d78c708f52b9cf30245b69b8
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/ruler/statefulset-ruler.yaml

*** /home/runner/work/loki/loki/output/base/default-single-binary-values/loki/templates/ruler/statefulset-ruler.yaml	2024-09-17 14:52:24.366725800 +0000
--- /home/runner/work/loki/loki/output/pr/default-single-binary-values/loki/templates/ruler/statefulset-ruler.yaml	2024-09-17 14:52:26.466752946 +0000
***************
*** 24,30 ****
 template:
 metadata:
 annotations:
! checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 24,30 ----
 template:
 metadata:
 annotations:
! checksum/config: 46367dc8cfea5f01466a79c988674cf14664a9c6d78c708f52b9cf30245b69b8
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

Removed Files

loki/templates/bloom-builder/service-bloom-builder.yaml

---
# Source: loki/templates/bloom-builder/service-bloom-builder.yaml
apiVersion: v1
kind: Service
metadata:
 name: loki-test-chart-name-bloom-builder
 namespace: default
 labels:
 helm.sh/chart: loki-6.12.0
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
 app.kubernetes.io/version: "3.1.1"
 app.kubernetes.io/managed-by: Helm
 app.kubernetes.io/component: bloom-builder
spec:
 type: ClusterIP
 publishNotReadyAddresses: true
 ports:
 - name: http-metrics
 port: 3100
 targetPort: http-metrics
 protocol: TCP
 - name: grpc
 port: 9095
 targetPort: grpc
 protocol: TCP
 - name: grpclb
 port: 9096
 targetPort: grpc
 protocol: TCP
 selector:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
 app.kubernetes.io/component: bloom-builder

loki/templates/bloom-builder/deployment-bloom-builder.yaml

---
# Source: loki/templates/bloom-builder/deployment-bloom-builder.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
 name: loki-test-chart-name-bloom-builder
 namespace: default
 labels:
 helm.sh/chart: loki-6.12.0
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
 app.kubernetes.io/version: "3.1.1"
 app.kubernetes.io/managed-by: Helm
 app.kubernetes.io/component: bloom-builder
spec:
 replicas: 0
 strategy:
 rollingUpdate:
 maxSurge: 0
 maxUnavailable: 1
 revisionHistoryLimit: 10
 selector:
 matchLabels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
 app.kubernetes.io/component: bloom-builder
 template:
 metadata:
 annotations:
 checksum/config: 927412fe484bf5103a8ac7034f7bb86d52494eaff45e6aaf2ab9e5f9076b436d
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
 app.kubernetes.io/component: bloom-builder
 app.kubernetes.io/part-of: memberlist
 spec:
 serviceAccountName: loki
 
 securityContext:
 fsGroup: 10001
 runAsGroup: 10001
 runAsNonRoot: true
 runAsUser: 10001
 terminationGracePeriodSeconds: 30
 containers:
 - name: bloom-builder
 image: docker.io/grafana/loki:3.1.1
 imagePullPolicy: IfNotPresent
 args:
 - -config.file=/etc/loki/config/config.yaml
 - -target=bloom-builder
 ports:
 - name: http-metrics
 containerPort: 3100
 protocol: TCP
 - name: grpc
 containerPort: 9095
 protocol: TCP
 - name: http-memberlist
 containerPort: 7946
 protocol: TCP
 securityContext:
 allowPrivilegeEscalation: false
 capabilities:
 drop:
 - ALL
 readOnlyRootFilesystem: true
 readinessProbe:
 httpGet:
 path: /ready
 port: http-metrics
 initialDelaySeconds: 30
 timeoutSeconds: 1
 volumeMounts:
 - name: config
 mountPath: /etc/loki/config
 - name: runtime-config
 mountPath: /etc/loki/runtime-config
 resources:
 {}
 affinity:
 podAntiAffinity:
 requiredDuringSchedulingIgnoredDuringExecution:
 - labelSelector:
 matchLabels:
 app.kubernetes.io/component: bloom-builder
 topologyKey: kubernetes.io/hostname
 volumes:
 - name: config
 configMap:
 name: loki
 items:
 - key: "config.yaml"
 path: "config.yaml"
 - name: runtime-config
 configMap:
 name: loki-runtime

loki/templates/bloom-builder/service-bloom-builder-headless.yaml

---
# Source: loki/templates/bloom-builder/service-bloom-builder-headless.yaml
apiVersion: v1
kind: Service
metadata:
 name: loki-test-chart-name-bloom-builder-headless
 namespace: default
 labels:
 helm.sh/chart: loki-6.12.0
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
 app.kubernetes.io/version: "3.1.1"
 app.kubernetes.io/managed-by: Helm
 app.kubernetes.io/component: bloom-builder
 prometheus.io/service-monitor: "false"
spec:
 clusterIP: None
 type: ClusterIP
 publishNotReadyAddresses: true
 ports:
 - name: http-metrics
 port: 3100
 targetPort: http-metrics
 protocol: TCP
 - name: grpc
 port: 9095
 targetPort: grpc
 protocol: TCP
 - name: grpclb
 port: 9096
 targetPort: grpc
 protocol: TCP
 selector:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
 app.kubernetes.io/component: bloom-builder

Scenario: default-values (Added: 0, Modified: 4, Removed: 0)

Summary:

Added: 0
Modified: 4
Removed: 0

Added Files

No added files

Modified Files

loki/templates/read/deployment-read.yaml

*** /home/runner/work/loki/loki/output/base/default-values/loki/templates/read/deployment-read.yaml	2024-09-17 14:52:24.458726989 +0000
--- /home/runner/work/loki/loki/output/pr/default-values/loki/templates/read/deployment-read.yaml	2024-09-17 14:52:26.558754135 +0000
***************
*** 28,34 ****
 template:
 metadata:
 annotations:
! checksum/config: 500ba3df68a94b828b3eb91169c199117adfeef5370bb54ec4d7d3ca3536b480
 labels:
 app.kubernetes.io/part-of: memberlist
 app.kubernetes.io/name: loki
--- 28,34 ----
 template:
 metadata:
 annotations:
! checksum/config: dcd7772b5baa16c9a94cbfde94aaf405e56672ed8265b9b70867dc3c97127fcd
 labels:
 app.kubernetes.io/part-of: memberlist
 app.kubernetes.io/name: loki

loki/templates/config.yaml

*** /home/runner/work/loki/loki/output/base/default-values/loki/templates/config.yaml	2024-09-17 14:52:24.458726989 +0000
--- /home/runner/work/loki/loki/output/pr/default-values/loki/templates/config.yaml	2024-09-17 14:52:26.558754135 +0000
***************
*** 15,20 ****
--- 15,28 ----
 config.yaml: |
 
 auth_enabled: true
+ bloom_build:
+ builder:
+ planner_address: loki-backend-headless.default.svc.cluster.local:9095
+ enabled: false
+ bloom_gateway:
+ client:
+ addresses: dnssrvnoa+_grpc._tcp.loki-backend-headless.default.svc.cluster.local
+ enabled: false
 chunk_store_config:
 chunk_cache_config:
 background:
***************
*** 97,102 ****
--- 105,112 ----
 http_server_read_timeout: 600s
 http_server_write_timeout: 600s
 storage_config:
+ bloom_shipper:
+ working_directory: /var/loki/data/bloomshipper
 boltdb_shipper:
 index_gateway_client:
 server_address: dns+loki-backend-headless.default.svc.cluster.local:9095

loki/templates/write/statefulset-write.yaml

*** /home/runner/work/loki/loki/output/base/default-values/loki/templates/write/statefulset-write.yaml	2024-09-17 14:52:24.458726989 +0000
--- /home/runner/work/loki/loki/output/pr/default-values/loki/templates/write/statefulset-write.yaml	2024-09-17 14:52:26.558754135 +0000
***************
*** 29,35 ****
 template:
 metadata:
 annotations:
! checksum/config: 500ba3df68a94b828b3eb91169c199117adfeef5370bb54ec4d7d3ca3536b480
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 29,35 ----
 template:
 metadata:
 annotations:
! checksum/config: dcd7772b5baa16c9a94cbfde94aaf405e56672ed8265b9b70867dc3c97127fcd
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/backend/statefulset-backend.yaml

*** /home/runner/work/loki/loki/output/base/default-values/loki/templates/backend/statefulset-backend.yaml	2024-09-17 14:52:24.458726989 +0000
--- /home/runner/work/loki/loki/output/pr/default-values/loki/templates/backend/statefulset-backend.yaml	2024-09-17 14:52:26.558754135 +0000
***************
*** 33,39 ****
 template:
 metadata:
 annotations:
! checksum/config: 500ba3df68a94b828b3eb91169c199117adfeef5370bb54ec4d7d3ca3536b480
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 33,39 ----
 template:
 metadata:
 annotations:
! checksum/config: dcd7772b5baa16c9a94cbfde94aaf405e56672ed8265b9b70867dc3c97127fcd
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

Removed Files

No removed files

Scenario: ingress-values (Added: 0, Modified: 4, Removed: 0)

Summary:

Added: 0
Modified: 4
Removed: 0

Added Files

No added files

Modified Files

loki/templates/read/deployment-read.yaml

*** /home/runner/work/loki/loki/output/base/ingress-values/loki/templates/read/deployment-read.yaml	2024-09-17 14:52:24.550728179 +0000
--- /home/runner/work/loki/loki/output/pr/ingress-values/loki/templates/read/deployment-read.yaml	2024-09-17 14:52:26.654755377 +0000
***************
*** 28,34 ****
 template:
 metadata:
 annotations:
! checksum/config: 500ba3df68a94b828b3eb91169c199117adfeef5370bb54ec4d7d3ca3536b480
 labels:
 app.kubernetes.io/part-of: memberlist
 app.kubernetes.io/name: loki
--- 28,34 ----
 template:
 metadata:
 annotations:
! checksum/config: dcd7772b5baa16c9a94cbfde94aaf405e56672ed8265b9b70867dc3c97127fcd
 labels:
 app.kubernetes.io/part-of: memberlist
 app.kubernetes.io/name: loki

loki/templates/config.yaml

*** /home/runner/work/loki/loki/output/base/ingress-values/loki/templates/config.yaml	2024-09-17 14:52:24.550728179 +0000
--- /home/runner/work/loki/loki/output/pr/ingress-values/loki/templates/config.yaml	2024-09-17 14:52:26.654755377 +0000
***************
*** 15,20 ****
--- 15,28 ----
 config.yaml: |
 
 auth_enabled: true
+ bloom_build:
+ builder:
+ planner_address: loki-backend-headless.default.svc.cluster.local:9095
+ enabled: false
+ bloom_gateway:
+ client:
+ addresses: dnssrvnoa+_grpc._tcp.loki-backend-headless.default.svc.cluster.local
+ enabled: false
 chunk_store_config:
 chunk_cache_config:
 background:
***************
*** 97,102 ****
--- 105,112 ----
 http_server_read_timeout: 600s
 http_server_write_timeout: 600s
 storage_config:
+ bloom_shipper:
+ working_directory: /var/loki/data/bloomshipper
 boltdb_shipper:
 index_gateway_client:
 server_address: dns+loki-backend-headless.default.svc.cluster.local:9095

loki/templates/write/statefulset-write.yaml

*** /home/runner/work/loki/loki/output/base/ingress-values/loki/templates/write/statefulset-write.yaml	2024-09-17 14:52:24.550728179 +0000
--- /home/runner/work/loki/loki/output/pr/ingress-values/loki/templates/write/statefulset-write.yaml	2024-09-17 14:52:26.654755377 +0000
***************
*** 29,35 ****
 template:
 metadata:
 annotations:
! checksum/config: 500ba3df68a94b828b3eb91169c199117adfeef5370bb54ec4d7d3ca3536b480
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 29,35 ----
 template:
 metadata:
 annotations:
! checksum/config: dcd7772b5baa16c9a94cbfde94aaf405e56672ed8265b9b70867dc3c97127fcd
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/backend/statefulset-backend.yaml

*** /home/runner/work/loki/loki/output/base/ingress-values/loki/templates/backend/statefulset-backend.yaml	2024-09-17 14:52:24.550728179 +0000
--- /home/runner/work/loki/loki/output/pr/ingress-values/loki/templates/backend/statefulset-backend.yaml	2024-09-17 14:52:26.654755377 +0000
***************
*** 33,39 ****
 template:
 metadata:
 annotations:
! checksum/config: 500ba3df68a94b828b3eb91169c199117adfeef5370bb54ec4d7d3ca3536b480
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 33,39 ----
 template:
 metadata:
 annotations:
! checksum/config: dcd7772b5baa16c9a94cbfde94aaf405e56672ed8265b9b70867dc3c97127fcd
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

Removed Files

No removed files

Scenario: legacy-monitoring-values (Added: 0, Modified: 4, Removed: 0)

Summary:

Added: 0
Modified: 4
Removed: 0

Added Files

No added files

Modified Files

loki/templates/read/deployment-read.yaml

*** /home/runner/work/loki/loki/output/base/legacy-monitoring-values/loki/templates/read/deployment-read.yaml	2024-09-17 14:52:24.650729471 +0000
--- /home/runner/work/loki/loki/output/pr/legacy-monitoring-values/loki/templates/read/deployment-read.yaml	2024-09-17 14:52:26.758756721 +0000
***************
*** 28,34 ****
 template:
 metadata:
 annotations:
! checksum/config: 500ba3df68a94b828b3eb91169c199117adfeef5370bb54ec4d7d3ca3536b480
 labels:
 app.kubernetes.io/part-of: memberlist
 app.kubernetes.io/name: loki
--- 28,34 ----
 template:
 metadata:
 annotations:
! checksum/config: dcd7772b5baa16c9a94cbfde94aaf405e56672ed8265b9b70867dc3c97127fcd
 labels:
 app.kubernetes.io/part-of: memberlist
 app.kubernetes.io/name: loki

loki/templates/config.yaml

*** /home/runner/work/loki/loki/output/base/legacy-monitoring-values/loki/templates/config.yaml	2024-09-17 14:52:24.650729471 +0000
--- /home/runner/work/loki/loki/output/pr/legacy-monitoring-values/loki/templates/config.yaml	2024-09-17 14:52:26.754756669 +0000
***************
*** 15,20 ****
--- 15,28 ----
 config.yaml: |
 
 auth_enabled: true
+ bloom_build:
+ builder:
+ planner_address: loki-backend-headless.default.svc.cluster.local:9095
+ enabled: false
+ bloom_gateway:
+ client:
+ addresses: dnssrvnoa+_grpc._tcp.loki-backend-headless.default.svc.cluster.local
+ enabled: false
 chunk_store_config:
 chunk_cache_config:
 background:
***************
*** 97,102 ****
--- 105,112 ----
 http_server_read_timeout: 600s
 http_server_write_timeout: 600s
 storage_config:
+ bloom_shipper:
+ working_directory: /var/loki/data/bloomshipper
 boltdb_shipper:
 index_gateway_client:
 server_address: dns+loki-backend-headless.default.svc.cluster.local:9095

loki/templates/write/statefulset-write.yaml

*** /home/runner/work/loki/loki/output/base/legacy-monitoring-values/loki/templates/write/statefulset-write.yaml	2024-09-17 14:52:24.650729471 +0000
--- /home/runner/work/loki/loki/output/pr/legacy-monitoring-values/loki/templates/write/statefulset-write.yaml	2024-09-17 14:52:26.758756721 +0000
***************
*** 29,35 ****
 template:
 metadata:
 annotations:
! checksum/config: 500ba3df68a94b828b3eb91169c199117adfeef5370bb54ec4d7d3ca3536b480
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 29,35 ----
 template:
 metadata:
 annotations:
! checksum/config: dcd7772b5baa16c9a94cbfde94aaf405e56672ed8265b9b70867dc3c97127fcd
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/backend/statefulset-backend.yaml

*** /home/runner/work/loki/loki/output/base/legacy-monitoring-values/loki/templates/backend/statefulset-backend.yaml	2024-09-17 14:52:24.650729471 +0000
--- /home/runner/work/loki/loki/output/pr/legacy-monitoring-values/loki/templates/backend/statefulset-backend.yaml	2024-09-17 14:52:26.758756721 +0000
***************
*** 33,39 ****
 template:
 metadata:
 annotations:
! checksum/config: 500ba3df68a94b828b3eb91169c199117adfeef5370bb54ec4d7d3ca3536b480
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name
--- 33,39 ----
 template:
 metadata:
 annotations:
! checksum/config: dcd7772b5baa16c9a94cbfde94aaf405e56672ed8265b9b70867dc3c97127fcd
 labels:
 app.kubernetes.io/name: loki
 app.kubernetes.io/instance: loki-test-chart-name

Removed Files

No removed files

Scenario: simple-scalable-aws-kube-irsa-values (Added: 0, Modified: 6, Removed: 0)

Summary:

Added: 0
Modified: 6
Removed: 0

Added Files

No added files

Modified Files

loki/templates/admin-api/deployment-admin-api.yaml

*** /home/runner/work/loki/loki/output/base/simple-scalable-aws-kube-irsa-values/loki/templates/admin-api/deployment-admin-api.yaml	2024-09-17 14:52:24.754730815 +0000
--- /home/runner/work/loki/loki/output/pr/simple-scalable-aws-kube-irsa-values/loki/templates/admin-api/deployment-admin-api.yaml	2024-09-17 14:52:26.862758065 +0000
***************
*** 33,39 ****
 target: admin-api
 app.kubernetes.io/part-of: memberlist
 annotations:
! checksum/config: 40ee874b8d9cd2ff500f549233cda3d88f452f13da79551155f43dbe7f2a0b65
 spec:
 serviceAccountName: enterprise-logs
 securityContext:
--- 33,39 ----
 target: admin-api
 app.kubernetes.io/part-of: memberlist
 annotations:
! checksum/config: 1c4ffb6d5d2060dcd93c506bd766b66dad54da9be377c9c129baf386794f8d00
 spec:
 serviceAccountName: enterprise-logs
 securityContext:

loki/templates/read/deployment-read.yaml

*** /home/runner/work/loki/loki/output/base/simple-scalable-aws-kube-irsa-values/loki/templates/read/deployment-read.yaml	2024-09-17 14:52:24.754730815 +0000
--- /home/runner/work/loki/loki/output/pr/simple-scalable-aws-kube-irsa-values/loki/templates/read/deployment-read.yaml	2024-09-17 14:52:26.862758065 +0000
***************
*** 28,34 ****
 template:
 metadata:
 annotations:
! checksum/config: 40ee874b8d9cd2ff500f549233cda3d88f452f13da79551155f43dbe7f2a0b65
 labels:
 app.kubernetes.io/part-of: memberlist
 app.kubernetes.io/name: enterprise-logs
--- 28,34 ----
 template:
 metadata:
 annotations:
! checksum/config: 1c4ffb6d5d2060dcd93c506bd766b66dad54da9be377c9c129baf386794f8d00
 labels:
 app.kubernetes.io/part-of: memberlist
 app.kubernetes.io/name: enterprise-logs

loki/templates/config.yaml

*** /home/runner/work/loki/loki/output/base/simple-scalable-aws-kube-irsa-values/loki/templates/config.yaml	2024-09-17 14:52:24.754730815 +0000
--- /home/runner/work/loki/loki/output/pr/simple-scalable-aws-kube-irsa-values/loki/templates/config.yaml	2024-09-17 14:52:26.862758065 +0000
***************
*** 22,27 ****
--- 22,35 ----
 auth:
 type: enterprise
 auth_enabled: true
+ bloom_build:
+ builder:
+ planner_address: enterprise-logs-backend-headless.default.svc.cluster.local:9095
+ enabled: false
+ bloom_gateway:
+ client:
+ addresses: dnssrvnoa+_grpc._tcp.enterprise-logs-backend-headless.default.svc.cluster.local
+ enabled: false
 chunk_store_config:
 chunk_cache_config:
 background:
***************
*** 109,114 ****
--- 117,124 ----
 http_server_read_timeout: 600s
 http_server_write_timeout: 600s
 storage_config:
+ bloom_shipper:
+ working_directory: /var/loki/data/bloomshipper
 boltdb_shipper:
 index_gateway_client:
 server_address: dns+enterprise-logs-backend-headless.default.svc.cluster.local:9095

loki/templates/gateway/deployment-gateway-enterprise.yaml

*** /home/runner/work/loki/loki/output/base/simple-scalable-aws-kube-irsa-values/loki/templates/gateway/deployment-gateway-enterprise.yaml	2024-09-17 14:52:24.754730815 +0000
--- /home/runner/work/loki/loki/output/pr/simple-scalable-aws-kube-irsa-values/loki/templates/gateway/deployment-gateway-enterprise.yaml	2024-09-17 14:52:26.862758065 +0000
***************
*** 28,34 ****
 app.kubernetes.io/instance: loki-test-chart-name
 app.kubernetes.io/component: gateway
 annotations:
! checksum/config: 40ee874b8d9cd2ff500f549233cda3d88f452f13da79551155f43dbe7f2a0b65
 spec:
 serviceAccountName: enterprise-logs
 securityContext:
--- 28,34 ----
 app.kubernetes.io/instance: loki-test-chart-name
 app.kubernetes.io/component: gateway
 annotations:
! checksum/config: 1c4ffb6d5d2060dcd93c506bd766b66dad54da9be377c9c129baf386794f8d00
 spec:
 serviceAccountName: enterprise-logs
 securityContext:

loki/templates/write/statefulset-write.yaml

*** /home/runner/work/loki/loki/output/base/simple-scalable-aws-kube-irsa-values/loki/templates/write/statefulset-write.yaml	2024-09-17 14:52:24.754730815 +0000
--- /home/runner/work/loki/loki/output/pr/simple-scalable-aws-kube-irsa-values/loki/templates/write/statefulset-write.yaml	2024-09-17 14:52:26.862758065 +0000
***************
*** 29,35 ****
 template:
 metadata:
 annotations:
! checksum/config: 40ee874b8d9cd2ff500f549233cda3d88f452f13da79551155f43dbe7f2a0b65
 labels:
 app.kubernetes.io/name: enterprise-logs
 app.kubernetes.io/instance: loki-test-chart-name
--- 29,35 ----
 template:
 metadata:
 annotations:
! checksum/config: 1c4ffb6d5d2060dcd93c506bd766b66dad54da9be377c9c129baf386794f8d00
 labels:
 app.kubernetes.io/name: enterprise-logs
 app.kubernetes.io/instance: loki-test-chart-name

loki/templates/backend/statefulset-backend.yaml

*** /home/runner/work/loki/loki/output/base/simple-scalable-aws-kube-irsa-values/loki/templates/backend/statefulset-backend.yaml	2024-09-17 14:52:24.754730815 +0000
--- /home/runner/work/loki/loki/output/pr/simple-scalable-aws-kube-irsa-values/loki/templates/backend/statefulset-backend.yaml	2024-09-17 14:52:26.862758065 +0000
***************
*** 33,39 ****
 template:
 metadata:
 annotations:
! checksum/config: 40ee874b8d9cd2ff500f549233cda3d88f452f13da79551155f43dbe7f2a0b65
 labels:
 app.kubernetes.io/name: enterprise-logs
 app.kubernetes.io/instance: loki-test-chart-name
--- 33,39 ----
 template:
 metadata:
 annotations:
! checksum/config: 1c4ffb6d5d2060dcd93c506bd766b66dad54da9be377c9c129baf386794f8d00
 labels:
 app.kubernetes.io/name: enterprise-logs
 app.kubernetes.io/instance: loki-test-chart-name

Removed Files

No removed files

trevorwhitney

LGTM, can we get a volunteer from the group that submitted bugs from the first PR to run this branch? We all run things a little different, and I'd like to make sure we're actually fixing the problem for those who reported it (since it wasn't a problem for us).

chaudum · 2024-09-17T14:59:53Z

LGTM, can we get a volunteer from the group that submitted bugs from the first PR to run this branch? We all run things a little different, and I'd like to make sure we're actually fixing the problem for those who reported it (since it wasn't a problem for us).

@diranged @z0rc We would very much appreciate if you could test this branch before we merge it. We want to ensure that we fix the mentioned issues.

chaudum requested a review from a team as a code owner September 12, 2024 11:44

pull-request-size bot added the size/L label Sep 12, 2024

github-actions bot added area/helm type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories labels Sep 12, 2024

chaudum mentioned this pull request Sep 12, 2024

feat(helm): Replace bloom compactor with bloom planner and builder #14003

Merged

7 tasks

chaudum force-pushed the chaudum/helm-fix-bloom-planner-builder-storage branch from 6204184 to e287c7c Compare September 12, 2024 14:30

chaudum added 4 commits September 12, 2024 16:58

Add required configuration for bloom builder and bloom gateway

6dfbc2c

Signed-off-by: Christian Haudum <[email protected]>

Fix: Configure emphemeral storage for bloom builder working directory

c13a5fa

Signed-off-by: Christian Haudum <[email protected]>

Fix: Configure PV claims for bloom gateway and bloom builder correctly

a9bd6c0

Signed-off-by: Christian Haudum <[email protected]>

Fix: Do not create bloom Deployment/Statefulset if replicas=0

6d45dbf

Signed-off-by: Christian Haudum <[email protected]>

chaudum force-pushed the chaudum/helm-fix-bloom-planner-builder-storage branch from e287c7c to 6d45dbf Compare September 12, 2024 14:58

diranged reviewed Sep 12, 2024

View reviewed changes

chaudum added 2 commits September 13, 2024 07:30

fixup! Fix: Do not create bloom Deployment/Statefulset if replicas=0

ae5085e

Signed-off-by: Christian Haudum <[email protected]>

Merge branch 'main' into chaudum/helm-fix-bloom-planner-builder-storage

00cb557

trevorwhitney approved these changes Sep 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(helm): Various fixes and enhancements for bloom components #14128

fix(helm): Various fixes and enhancements for bloom components #14128

chaudum commented Sep 12, 2024 •

edited

Loading

diranged Sep 12, 2024

chaudum Sep 12, 2024

diranged left a comment

github-actions bot commented Sep 17, 2024

Added Files

Modified Files

Removed Files

Added Files

Modified Files

Removed Files

Added Files

Modified Files

Removed Files

Added Files

Modified Files

Removed Files

Added Files

Modified Files

Removed Files

trevorwhitney left a comment

chaudum commented Sep 17, 2024

fix(helm): Various fixes and enhancements for bloom components #14128

Are you sure you want to change the base?

fix(helm): Various fixes and enhancements for bloom components #14128

Conversation

chaudum commented Sep 12, 2024 • edited Loading

diranged Sep 12, 2024

Choose a reason for hiding this comment

chaudum Sep 12, 2024

Choose a reason for hiding this comment

diranged left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 17, 2024

Kubernetes Manifest Diff Summary

Added Files

Modified Files

Removed Files

Added Files

Modified Files

Removed Files

Added Files

Modified Files

Removed Files

Added Files

Modified Files

Removed Files

Added Files

Modified Files

Removed Files

trevorwhitney left a comment

Choose a reason for hiding this comment

chaudum commented Sep 17, 2024

chaudum commented Sep 12, 2024 •

edited

Loading