Add option for configuring HPA with custom/external metrics (#467)

* feat: option for custom/external metrics HPA Signed-off-by: Loc Mai <[email protected]> * docs: update CHANGELOG.md Signed-off-by: Loc Mai <[email protected]> * fix: lint Signed-off-by: Loc Mai <[email protected]> * docs: re-gen README Signed-off-by: Loc Mai <[email protected]> * docs: update CHANGELOG Signed-off-by: Loc Mai <[email protected]> * docs: update README format Signed-off-by: Loc Mai <[email protected]> * fix: lint Signed-off-by: Loc Mai <[email protected]> * chore: remove example Signed-off-by: Loc Mai <[email protected]> * docs: shorten description Signed-off-by: Loc Mai <[email protected]> --------- Signed-off-by: Loc Mai <[email protected]>
cortexproject · Jun 27, 2023 · d335fe6 · d335fe6
1 parent 8d883fd
commit d335fe6
Show file tree

Hide file tree

Showing 9 changed files with 43 additions and 0 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -2,6 +2,7 @@
 
 ## master / unreleased
 
+* [ENHANCEMENT] Add option to autoscale on multiple metrics and custom metrics metrics #467
 * [DEPENDENCY] Update quay.io/cortexproject/cortex Docker tag to v1.15.0 #453
 * [DEPENDENCY] Update quay.io/cortexproject/cortex Docker tag to v1.15.1 #456
 * [DEPENDENCY] Update quay.io/cortexproject/cortex Docker tag to v1.15.2 #459

diff --git a/README.md b/README.md
@@ -266,6 +266,7 @@ Kubernetes: `^1.19.0-0`
 | distributor.&ZeroWidthSpace;annotations | object | `{}` |  |
 | distributor.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;behavior | object | `{}` | Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior |
 | distributor.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;enabled | bool | `false` | Creates a HorizontalPodAutoscaler for the distributor pods. |
+| distributor.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;extraMetrics | list | `[]` | Optional custom and external metrics for the distributor pods to scale on In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects |
 | distributor.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;maxReplicas | int | `30` |  |
 | distributor.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;minReplicas | int | `2` |  |
 | distributor.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;targetCPUUtilizationPercentage | int | `80` |  |
@@ -327,6 +328,7 @@ Kubernetes: `^1.19.0-0`
 | ingester.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;behavior.&ZeroWidthSpace;scaleDown.&ZeroWidthSpace;stabilizationWindowSeconds | int | `3600` | uses metrics from the past 1h to make scaleDown decisions |
 | ingester.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;behavior.&ZeroWidthSpace;scaleUp.&ZeroWidthSpace;policies | list | `[{"periodSeconds":1800,"type":"Pods","value":1}]` | This default scaleup policy allows adding 1 pod every 30 minutes. Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior |
 | ingester.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;enabled | bool | `false` |  |
+| ingester.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;extraMetrics | list | `[]` | Optional custom and external metrics for the ingester pods to scale on In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects |
 | ingester.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;maxReplicas | int | `30` |  |
 | ingester.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;minReplicas | int | `3` |  |
 | ingester.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;targetMemoryUtilizationPercentage | int | `80` |  |
@@ -425,6 +427,7 @@ Kubernetes: `^1.19.0-0`
 | nginx.&ZeroWidthSpace;annotations | object | `{}` |  |
 | nginx.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;behavior | object | `{}` | Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior |
 | nginx.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;enabled | bool | `false` | Creates a HorizontalPodAutoscaler for the nginx pods. |
+| nginx.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;extraMetrics | list | `[]` | Optional custom and external metrics for the nginx pods to scale on In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects |
 | nginx.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;maxReplicas | int | `30` |  |
 | nginx.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;minReplicas | int | `2` |  |
 | nginx.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;targetCPUUtilizationPercentage | int | `80` |  |
@@ -572,6 +575,7 @@ Kubernetes: `^1.19.0-0`
 | querier.&ZeroWidthSpace;annotations | object | `{}` |  |
 | querier.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;behavior | object | `{}` | Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior |
 | querier.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;enabled | bool | `false` | Creates a HorizontalPodAutoscaler for the querier pods. |
+| querier.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;extraMetrics | list | `[]` | Optional custom and external metrics for the querier pods to scale on In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects |
 | querier.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;maxReplicas | int | `30` |  |
 | querier.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;minReplicas | int | `2` |  |
 | querier.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;targetCPUUtilizationPercentage | int | `80` |  |
@@ -714,6 +718,7 @@ Kubernetes: `^1.19.0-0`
 | ruler.&ZeroWidthSpace;annotations | object | `{}` |  |
 | ruler.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;behavior | object | `{}` | Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior |
 | ruler.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;enabled | bool | `false` | Creates a HorizontalPodAutoscaler for the ruler. |
+| ruler.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;extraMetrics | list | `[]` | Optional custom and external metrics for the ruler pods to scale on In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects |
 | ruler.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;maxReplicas | int | `30` |  |
 | ruler.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;minReplicas | int | `2` |  |
 | ruler.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;targetCPUUtilizationPercentage | int | `80` |  |
@@ -792,6 +797,7 @@ Kubernetes: `^1.19.0-0`
 | store_gateway.&ZeroWidthSpace;annotations | object | `{}` |  |
 | store_gateway.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;behavior | object | `{}` | Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior |
 | store_gateway.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;enabled | bool | `false` |  |
+| store_gateway.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;extraMetrics | list | `[]` | Optional custom and external metrics for the store gateway pods to scale on In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects |
 | store_gateway.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;maxReplicas | int | `30` |  |
 | store_gateway.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;minReplicas | int | `3` |  |
 | store_gateway.&ZeroWidthSpace;autoscaling.&ZeroWidthSpace;targetMemoryUtilizationPercentage | int | `80` |  |

diff --git a/templates/distributor/distributor-hpa.yaml b/templates/distributor/distributor-hpa.yaml
@@ -31,6 +31,9 @@ spec:
           type: Utilization
           averageUtilization: {{ . }}
   {{- end }}
+  {{- with .extraMetrics }}
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   {{- with .behavior }}
   behavior:
     {{- toYaml . | nindent 4 }}

diff --git a/templates/ingester/ingester-hpa.yaml b/templates/ingester/ingester-hpa.yaml
@@ -21,6 +21,9 @@ spec:
         target:
           type: Utilization
           averageUtilization: {{ .targetMemoryUtilizationPercentage }}
+  {{- with .extraMetrics }}
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   {{- with .behavior }}
   behavior:
     {{- toYaml . | nindent 4 }}

diff --git a/templates/nginx/nginx-hpa.yaml b/templates/nginx/nginx-hpa.yaml
@@ -31,6 +31,9 @@ spec:
           type: Utilization
           averageUtilization: {{ . }}
   {{- end }}
+  {{- with .extraMetrics }}
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   {{- with .behavior }}
   behavior:
     {{- toYaml . | nindent 4 }}

diff --git a/templates/querier/querier-hpa.yaml b/templates/querier/querier-hpa.yaml
@@ -31,6 +31,9 @@ spec:
           type: Utilization
           averageUtilization: {{ . }}
   {{- end }}
+  {{- with .extraMetrics }}
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   {{- with .behavior }}
   behavior:
     {{- toYaml . | nindent 4 }}

diff --git a/templates/ruler/ruler-hpa.yaml b/templates/ruler/ruler-hpa.yaml
@@ -31,6 +31,9 @@ spec:
           type: Utilization
           averageUtilization: {{ . }}
   {{- end }}
+  {{- with .extraMetrics }}
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   {{- with .behavior }}
   behavior:
     {{- toYaml . | nindent 4 }}

diff --git a/templates/store-gateway/store-gateway-hpa.yaml b/templates/store-gateway/store-gateway-hpa.yaml
@@ -21,6 +21,9 @@ spec:
         target:
           type: Utilization
           averageUtilization: {{ .targetMemoryUtilizationPercentage }}
+  {{- with .extraMetrics }}
+    {{- toYaml . | nindent 4 }}
+  {{- end }}
   {{- with .behavior }}
   behavior:
     {{- toYaml . | nindent 4 }}

diff --git a/values.yaml b/values.yaml
@@ -372,6 +372,9 @@ distributor:
     targetMemoryUtilizationPercentage: 0  # 80
     # -- Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior
     behavior: {}
+    # -- Optional custom and external metrics for the distributor pods to scale on
+    # In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects
+    extraMetrics: []
 
   persistentVolume:
     subPath:
@@ -496,6 +499,9 @@ ingester:
           - type: Pods
             value: 1
             periodSeconds: 1800
+    # -- Optional custom and external metrics for the ingester pods to scale on
+    # In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects
+    extraMetrics: []
 
   lifecycle:
     # -- The /shutdown preStop hook is recommended as part of the ingester
@@ -624,6 +630,9 @@ ruler:
     targetMemoryUtilizationPercentage: 80
     # -- Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior
     behavior: {}
+    # -- Optional custom and external metrics for the ruler pods to scale on
+    # In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects
+    extraMetrics: []
 
   validation:
     # -- Checks that the ruler is compatible with horizontal scaling,
@@ -773,6 +782,9 @@ querier:
     targetMemoryUtilizationPercentage: 0  # 80
     # -- Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior
     behavior: {}
+    # -- Optional custom and external metrics for the querier pods to scale on
+    # In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects
+    extraMetrics: []
 
   persistentVolume:
     subPath:
@@ -1293,6 +1305,9 @@ nginx:
     targetMemoryUtilizationPercentage: 0  # 80
     # -- Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior
     behavior: {}
+    # -- Optional custom and external metrics for the nginx pods to scale on
+    # In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects
+    extraMetrics: []
 
 store_gateway:
   enabled: true
@@ -1354,6 +1369,9 @@ store_gateway:
     targetMemoryUtilizationPercentage: 80
     # -- Ref: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-configurable-scaling-behavior
     behavior: {}
+    # -- Optional custom and external metrics for the store gateway pods to scale on
+    # In order to use this option , define a list of of specific following https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-multiple-metrics-and-custom-metrics and https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale-walkthrough/#autoscaling-on-metrics-not-related-to-kubernetes-objects
+    extraMetrics: []
 
   persistentVolume:
     # -- If true Store-gateway will create/use a Persistent Volume Claim