2.12.0

This release contains 531 PRs from 60 authors, including new contributors Benoit Schipper, Derek Cadzow, Edwin, Itay Kalfon, Ivan Farré Vicente, Jan O. Rundshagen, Jorge Turrado Ferrero, Lukas Monkevicius, Mickaël Canévet, Rafael Sathler, Rajakavitha Kodhandapani, Tim Kotowski, Vladimir Varankin, Zach, Zach Day, Zirko, blut, github-actions[bot], ncharaf, zhehao-grafana. Thank you!

Grafana Mimir version 2.12.0 release notes

Grafana Labs is excited to announce version 2.12 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

• Added support to only count series that are considered active through the Cardinality API endpoint /api/v1/cardinality/label_names by passing the count_method parameter.
  If set to active it counts only series that are considered active according to the -ingester.active-series-metrics-idle-timeout flag setting rather than counting all in-memory series.

• The "Store-gateway: bucket tenant blocks" admin page contains a new column "No Compact".
  If block no compaction marker is set, it specifies the reason and the date the marker is added.

• The estimated number of compaction jobs based on the current bucket-index is now computed by the compactor.
  The result is tracked by the new cortex_bucket_index_compaction_jobs metric.
  If this computation fails, the cortex_bucket_index_compaction_jobs_errors_total metric is updated instead.
  The estimated number of compaction jobs is also shown in Top tenants, Tenants, and Compactor dashboards.

• Added mimir-distroless container image built upon a distroless image (gcr.io/distroless/static-debian12).
  This improvement minimizes attack surfaces and potential CVEs by trimming down the dependencies within the image.
  After comprehensive testing, the Mimir maintainers plan to shift from the current image to the distroless version.

Additionally, the following previously experimental features are now considered stable:

• The number of pre-allocated workers used to forward push requests to the ingesters, configurable via the -distributor.reusable-ingester-push-workers CLI flag on distributors.
  It now defaults to 2000.
  Note that this is a performance optimization, and not a limiting feature.
  If not enough workers available, new goroutines will be spawned.

• The number of gRPC server workers used to serve the requests, configurable via the -server.grpc.num-workers CLI flag.
  It now defaults to 100.
  Note that this is the number of pre-allocated long-lived workers, and not a limiting feature.
  If not enough workers are available, new goroutines will be spawned.

• The maximum number of concurrent index header loads across all tenants, configurable via the -blocks-storage.bucket-store.index-header.lazy-loading-concurrency CLI flag on store-gateways.
  It defaults to 4.

• The maximum time to wait for the query-frontend to become ready before rejecting requests, configurable via the -query-frontend.not-running-timeout CLI flag on query-frontends.
  It now defaults to 2s.

• The CLI flag that allows queriers to reduce pressure on ingesters by initially querying only the minimum set of ingesters required to reach quorum, -querier.minimize-ingester-requests.
  It is now enabled by default.

• Spread-minimizing token-related CLI flags: -ingester.ring.token-generation-strategy, -ingester.ring.spread-minimizing-zones and -ingester.ring.spread-minimizing-join-ring-in-order.
  You can read more about this feature in our blog post.

Important changes

In Grafana Mimir 2.12 the following behavior has changed:

• Store-gateway now persists a sparse version of the index-header to disk on construction and loads sparse index-headers from disk instead of the whole index-header.
  This improves the speed at which index headers are lazy-loaded from disk by up to 90%. The added disk usage is in the order of 1-2%.

• Alertmanager deprecated the v1 API. All v1 API endpoints now respond with a JSON deprecation notice and a status code of 410.
  All endpoints have a v2 equivalent.
  The list of endpoints is:

  • <alertmanager-web.external-url>/api/v1/alerts
  • <alertmanager-web.external-url>/api/v1/receivers
  • <alertmanager-web.external-url>/api/v1/silence/{id}
  • <alertmanager-web.external-url>/api/v1/silences
  • <alertmanager-web.external-url>/api/v1/status

• Exemplar's label traceID has been changed to trace_id to be consistent with the OpenTelemetry standard.

• Errors returned by ingesters now contain only gRPC status codes.
  Previously they contained both gRPC and HTTP status codes.

  {{< admonition type="warning" >}}
  To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
  Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.
  {{< /admonition >}}

• Responses with gRPC status codes are now reported as status_code labels in the cortex_request_duration_seconds and cortex_ingester_client_request_duration_seconds metrics.

• Responses with HTTP 4xx status codes are now treated as errors and used in status_code label of request duration metric.

The default value of the following CLI flags have been changed:

• -blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
• -blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
• -blocks-storage.bucket-store.tenant-sync-concurrency from 10 to 1.
• -query-frontend.max-cache-freshness from 1m to 10m.
• -distributor.write-requests-buffer-pooling-enabled from false to true.
• -locks-storage.bucket-store.block-sync-concurrency from 20 to 4.
• -memberlist.stream-timeout from 10s to 2s.
• -server.report-grpc-codes-in-instrumentation-label-enabled from false to true.

The following deprecated configuration options are removed in Grafana Mimir 2.12:

• The YAML setting frontend.cache_unaligned_requests.
• Experimental CLI flag -querier.prefer-streaming-chunks-from-ingesters.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.14:

• The CLI flag -ingester.limit-inflight-requests-using-grpc-method-limiter.
  It now defaults to true.

• The CLI flag -ingester.return-only-grpc-errors.
  It now defaults to true.

  {{< admonition type="warning" >}}
  To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
  Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.
  {{< /admonition >}}

• The CLI flag -ingester.client.report-grpc-codes-in-instrumentation-label-enabled.
  It now defaults to true.

• The CLI flag -distributor.limit-inflight-requests-using-grpc-method-limiter.
  It now defaults to true.

• The CLI flag -distributor.enable-otlp-metadata-storage.
  It now defaults to true.

• The CLI flag -querier.max-query-into-future.

The following metrics are removed or deprecated:

• cortex_bucket_store_blocks_loaded_by_duration has been removed.
• cortex_distributor_sample_delay_seconds has been deprecated and will be removed in Mimir 2.14.

Experimental features

Grafana Mimir 2.12 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

• The maximum number of tenant IDs that may be for a federated query can be configured via the -tenant-federation.max-tenants CLI flag on query-frontends.
  By default, it's 0, meaning that the limit is disabled.

• Sharding of active series queries can be enabled via the -query-frontend.shard-active-series-queries CLI flag on query-frontends.

• Timely head compaction can be enabled via the -blocks-storage.tsdb.timely-head-compaction-enabled on ingesters.
  If enabled, the head compaction happens when the min block range can no longer be appended, without requiring 1.5x the chunk range worth of data in the head.

• Streaming of responses from querier to query-frontend can be enabled via the -querier.response-streaming-enabled CLI flag on queriers.
  This is currently supported only for responses from the /api/v1/cardinality/active_series endpoint.

• The maximum response size for active series queries, in bytes, can be set via the -querier.active-series-results-max-size-bytes CLI flag on queriers.

• Metric relabeling on a per-tenant basis can be forcefully disabled via the -distributor.metric-relabeling-enabled CLI flag on rulers.
  Metrics relabeling is enabled by default.

• Query Queue Load Balancing by Query Component. Tenant query queues in the query-scheduler can now be split into subqueues by which query component is expected to be utilized to complete the query: ingesters, store-gateways, both, or uncategorized.
  Dequeuing queries for a given tenant will rotate through the query component subqueues via simple round-robin.
  In the event that the one of the query components (ingesters or store-gateways) experience a slowdown, queries only utilizing the other query component can continue to be serviced.
  This feature is r...

2.12.0-rc.1

This release contains 525 PRs from 60 authors, including new contributors Benoit Schipper, Derek Cadzow, Edwin, Itay Kalfon, Ivan Farré Vicente, Jan O. Rundshagen, Jorge Turrado Ferrero, Lukas Monkevicius, Mickaël Canévet, Rafael Sathler, Rajakavitha Kodhandapani, Tim Kotowski, Vladimir Varankin, Zach, Zach Day, Zirko, blut, github-actions[bot], ncharaf, zhehao-grafana. Thank you!

Grafana Mimir version 2.12.0-rc.1 release notes

Grafana Labs is excited to announce version 2.12 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

• Added support to only count series that are considered active through the Cardinality API endpoint /api/v1/cardinality/label_names by passing the count_method parameter.
  If set to active it counts only series that are considered active according to the -ingester.active-series-metrics-idle-timeout flag setting rather than counting all in-memory series.

• The "Store-gateway: bucket tenant blocks" admin page contains a new column "No Compact".
  If block no compaction marker is set, it specifies the reason and the date the marker is added.

• The estimated number of compaction jobs based on the current bucket-index is now computed by the compactor.
  The result is tracked by the new cortex_bucket_index_compaction_jobs metric.
  If this computation fails, the cortex_bucket_index_compaction_jobs_errors_total metric is updated instead.
  The estimated number of compaction jobs is also shown in Top tenants, Tenants, and Compactor dashboards.

• Added mimir-distroless container image built upon a distroless image (gcr.io/distroless/static-debian12).
  This improvement minimizes attack surfaces and potential CVEs by trimming down the dependencies within the image.
  After comprehensive testing, the Mimir maintainers plan to shift from the current image to the distroless version.

Additionally, the following previously experimental features are now considered stable:

• The number of pre-allocated workers used to forward push requests to the ingesters, configurable via the -distributor.reusable-ingester-push-workers CLI flag on distributors.
  It now defaults to 2000.
  Note that this is a performance optimization, and not a limiting feature.
  If not enough workers available, new goroutines will be spawned.

• The number of gRPC server workers used to serve the requests, configurable via the -server.grpc.num-workers CLI flag.
  It now defaults to 100.
  Note that this is the number of pre-allocated long-lived workers, and not a limiting feature.
  If not enough workers are available, new goroutines will be spawned.

• The maximum number of concurrent index header loads across all tenants, configurable via the -blocks-storage.bucket-store.index-header.lazy-loading-concurrency CLI flag on store-gateways.
  It defaults to 4.

• The maximum time to wait for the query-frontend to become ready before rejecting requests, configurable via the -query-frontend.not-running-timeout CLI flag on query-frontends.
  It now defaults to 2s.

• The CLI flag that allows queriers to reduce pressure on ingesters by initially querying only the minimum set of ingesters required to reach quorum, -querier.minimize-ingester-requests.
  It is now enabled by default.

• Spread-minimizing token-related CLI flags: -ingester.ring.token-generation-strategy, -ingester.ring.spread-minimizing-zones and -ingester.ring.spread-minimizing-join-ring-in-order.
  You can read more about this feature in our blog post.

Important changes

In Grafana Mimir 2.12 the following behavior has changed:

• Store-gateway now persists a sparse version of the index-header to disk on construction and loads sparse index-headers from disk instead of the whole index-header.
  This improves the speed at which index headers are lazy-loaded from disk by up to 90%. The added disk usage is in the order of 1-2%.

• Alertmanager deprecated the v1 API. All v1 API endpoints now respond with a JSON deprecation notice and a status code of 410.
  All endpoints have a v2 equivalent.
  The list of endpoints is:

  • <alertmanager-web.external-url>/api/v1/alerts
  • <alertmanager-web.external-url>/api/v1/receivers
  • <alertmanager-web.external-url>/api/v1/silence/{id}
  • <alertmanager-web.external-url>/api/v1/silences
  • <alertmanager-web.external-url>/api/v1/status

• Exemplar's label traceID has been changed to trace_id to be consistent with the OpenTelemetry standard.

• Errors returned by ingesters now contain only gRPC status codes.
  Previously they contained both gRPC and HTTP status codes.
  To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
  Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.

• Responses with gRPC status codes are now reported as status_code labels in the cortex_request_duration_seconds and cortex_ingester_client_request_duration_seconds metrics.

• Responses with HTTP 4xx status codes are now treated as errors and used in status_code label of request duration metric.

The default value of the following CLI flags have been changed:

• -blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
• -blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
• -blocks-storage.bucket-store.tenant-sync-concurrency from 10 to 1.
• -query-frontend.max-cache-freshness from 1m to 10m.
• -distributor.write-requests-buffer-pooling-enabled from false to true.
• -locks-storage.bucket-store.block-sync-concurrency from 20 to 4.
• -memberlist.stream-timeout from 10s to 2s.
• -server.report-grpc-codes-in-instrumentation-label-enabled from false to true.

The following deprecated configuration options are removed in Grafana Mimir 2.12:

• The YAML setting frontend.cache_unaligned_requests.
• Experimental CLI flag -querier.prefer-streaming-chunks-from-ingesters.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.14:

• The CLI flag -ingester.limit-inflight-requests-using-grpc-method-limiter.
  It now defaults to true.

• The CLI flag -ingester.return-only-grpc-errors.
  It now defaults to true.
  To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
  Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.

• The CLI flag -ingester.client.report-grpc-codes-in-instrumentation-label-enabled.
  It now defaults to true.

• The CLI flag -distributor.limit-inflight-requests-using-grpc-method-limiter.
  It now defaults to true.

• The CLI flag -distributor.enable-otlp-metadata-storage.
  It now defaults to true.

• The CLI flag -querier.max-query-into-future.

The following metrics are removed or deprecated:

• cortex_bucket_store_blocks_loaded_by_duration has been removed.
• cortex_distributor_sample_delay_seconds has been deprecated and will be removed in Mimir 2.14.

Experimental features

Grafana Mimir 2.12 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

• The maximum number of tenant IDs that may be for a federated query can be configured via the -tenant-federation.max-tenants CLI flag on query-frontends.
  By default, it's 0, meaning that the limit is disabled.

• Sharding of active series queries can be enabled via the -query-frontend.shard-active-series-queries CLI flag on query-frontends.

• Timely head compaction can be enabled via the -blocks-storage.tsdb.timely-head-compaction-enabled on ingesters.
  If enabled, the head compaction happens when the min block range can no longer be appended, without requiring 1.5x the chunk range worth of data in the head.

• Streaming of responses from querier to query-frontend can be enabled via the -querier.response-streaming-enabled CLI flag on queriers.
  This is currently supported only for responses from the /api/v1/cardinality/active_series endpoint.

• The maximum response size for active series queries, in bytes, can be set via the -querier.active-series-results-max-size-bytes CLI flag on queriers.

• Metric relabeling on a per-tenant basis can be forcefully disabled via the -distributor.metric-relabeling-enabled CLI flag on rulers.
  Metrics relabeling is enabled by default.

• Query Queue Load Balancing by Query Component. Tenant query queues in the query-scheduler can now be split into subqueues by which query component is expected to be utilized to complete the query: ingesters, store-gateways, both, or uncategorized.
  Dequeuing queries for a given tenant will rotate through the query component subqueues via simple round-robin.
  In the event that the one of the query components (ingesters or store-gateways) experience a slowdown, queries only utilizing the the other query component can continue to be serviced.
  This feature is recommended to be enabled.
  The following CLI flags must be set to true in order to be in effect:

  • `-query...

2.12.0-rc.0

This release contains 525 PRs from 60 authors, including new contributors Benoit Schipper, Derek Cadzow, Edwin, Itay Kalfon, Ivan Farré Vicente, Jan O. Rundshagen, Jorge Turrado Ferrero, Lukas Monkevicius, Mickaël Canévet, Rafael Sathler, Rajakavitha Kodhandapani, Tim Kotowski, Vladimir Varankin, Zach, Zach Day, Zirko, blut, github-actions[bot], ncharaf, zhehao-grafana. Thank you!

Grafana Mimir version 2.12.0-rc.0 release notes

Grafana Labs is excited to announce version 2.12 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bug fixes in this release.
For the complete list of changes, refer to the CHANGELOG.

Features and enhancements

• Added support to only count series that are considered active through the Cardinality API endpoint /api/v1/cardinality/label_names by passing the count_method parameter.
  If set to active it counts only series that are considered active according to the -ingester.active-series-metrics-idle-timeout flag setting rather than counting all in-memory series.

• The "Store-gateway: bucket tenant blocks" admin page contains a new column "No Compact".
  If block no compaction marker is set, it specifies the reason and the date the marker is added.

• The estimated number of compaction jobs based on the current bucket-index is now computed by the compactor.
  The result is tracked by the new cortex_bucket_index_compaction_jobs metric.
  If this computation fails, the cortex_bucket_index_compaction_jobs_errors_total metric is updated instead.
  The estimated number of compaction jobs is also shown in Top tenants, Tenants, and Compactor dashboards.

• Added mimir-distroless container image built upon a distroless image (gcr.io/distroless/static-debian12).
  This improvement minimizes attack surfaces and potential CVEs by trimming down the dependencies within the image.
  After comprehensive testing, the Mimir maintainers plan to shift from the current image to the distroless version.

Additionally, the following previously experimental features are now considered stable:

• The number of pre-allocated workers used to forward push requests to the ingesters, configurable via the -distributor.reusable-ingester-push-workers CLI flag on distributors.
  It now defaults to 2000.
  Note that this is a performance optimization, and not a limiting feature.
  If not enough workers available, new goroutines will be spawned.

• The number of gRPC server workers used to serve the requests, configurable via the -server.grpc.num-workers CLI flag.
  It now defaults to 100.
  Note that this is the number of pre-allocated long-lived workers, and not a limiting feature.
  If not enough workers are available, new goroutines will be spawned.

• The maximum number of concurrent index header loads across all tenants, configurable via the -blocks-storage.bucket-store.index-header.lazy-loading-concurrency CLI flag on store-gateways.
  It defaults to 4.

• The maximum time to wait for the query-frontend to become ready before rejecting requests, configurable via the -query-frontend.not-running-timeout CLI flags on query-frontends.
  It now defaults to 2s.

• Spread-minimizing token-related CLI flags: -ingester.ring.token-generation-strategy, -ingester.ring.spread-minimizing-zones and -ingester.ring.spread-minimizing-join-ring-in-order.
  You can read more about this feature in our blog post.

Important changes

In Grafana Mimir 2.12 the following behavior has changed:

• Store-gateway now persists a sparse version of the index-header to disk on construction and loads sparse index-headers from disk instead of the whole index-header.
  This improves the speed at which index headers are lazy-loaded from disk by up to 90%. The added disk usage is in the order of 1-2%.

• Alertmanager deprecated the v1 API. All v1 API endpoints now respond with a JSON deprecation notice and a status code of 410.
  All endpoints have a v2 equivalent.
  The list of endpoints is:

  • <alertmanager-web.external-url>/api/v1/alerts
  • <alertmanager-web.external-url>/api/v1/receivers
  • <alertmanager-web.external-url>/api/v1/silence/{id}
  • <alertmanager-web.external-url>/api/v1/silences
  • <alertmanager-web.external-url>/api/v1/status

• Exemplar's label traceID has been changed to trace_id to be consistent with the OpenTelemetry standard.

• Errors returned by ingesters now contain only gRPC status codes.
  Previously they contained both gRPC and HTTP status codes.
  To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
  Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.

• Responses with gRPC status codes are now reported as status_code labels in the cortex_request_duration_seconds and cortex_ingester_client_request_duration_seconds metrics.

• Responses with HTTP 4xx status codes are now treated as errors and used in status_code label of request duration metric.

The default value of the following CLI flags have been changed:

• -blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
• -blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes from 10MB to 100MB.
• -blocks-storage.bucket-store.tenant-sync-concurrency from 10 to 1.
• -query-frontend.max-cache-freshness from 1m to 10m.
• -distributor.write-requests-buffer-pooling-enabled from false to true.
• -locks-storage.bucket-store.block-sync-concurrency from 20 to 4.
• -memberlist.stream-timeout from 10s to 2s.
• -server.report-grpc-codes-in-instrumentation-label-enabled from false to true.

The following deprecated configuration options are removed in Grafana Mimir 2.12:

• The YAML setting frontend.cache_unaligned_requests.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.14:

• The CLI flag -ingester.limit-inflight-requests-using-grpc-method-limiter.
  It now defaults to true.

• The CLI flag -ingester.return-only-grpc-errors.
  It now defaults to true.
  To guarantee backwards compatibility when migrating from a version prior to 2.11, it's necessary to first migrate to version 2.11, and then to version 2.12.
  Otherwise, it might happen that during the migration, some ingester errors with HTTP status code 4xx won't be recognized, and the corresponding request will be repeated.

• The CLI flag -ingester.client.report-grpc-codes-in-instrumentation-label-enabled.
  It now defaults to true.

• The CLI flag -distributor.limit-inflight-requests-using-grpc-method-limiter.
  It now defaults to true.

• The CLI flag -distributor.enable-otlp-metadata-storage.
  It now defaults to true.

• The CLI flag -querier.max-query-into-future.

The following metrics are removed or deprecated:

• cortex_bucket_store_blocks_loaded_by_duration has been removed.
• cortex_distributor_sample_delay_seconds has been deprecated and will be removed in Mimir 2.14.

Experimental features

Grafana Mimir 2.12 includes new features that are considered experimental and disabled by default.
Use them with caution and report any issues you encounter:

• The maximum number of tenant IDs that may be for a federated query can be configured via the -tenant-federation.max-tenants CLI flag on query-frontends.
  By default, it's 0, meaning that the limit is disabled.

• Sharding of active series queries can be enabled via the -query-frontend.shard-active-series-queries CLI flag on query-frontends.

• Timely head compaction can be enabled via the -blocks-storage.tsdb.timely-head-compaction-enabled on ingesters.
  If enabled, the head compaction happens when the min block range can no longer be appended, without requiring 1.5x the chunk range worth of data in the head.

• Streaming of responses from querier to query-frontend can be enabled via the -querier.response-streaming-enabled CLI flag on queriers.
  This is currently supported only for responses from the /api/v1/cardinality/active_series endpoint.

• The maximum response size for active series queries, in bytes, can be set via the -querier.active-series-results-max-size-bytes CLI flag on queriers.

• Metric relabeling on a per-tenant basis can be forcefully disabled via the -distributor.metric-relabeling-enabled CLI flag on rulers.
  Metrics relabeling is enabled by default.

• Query Queue Load Balancing by Query Component. Tenant query queues in the query-scheduler can now be split into subqueues by which query component is expected to be utilized to complete the query: ingesters, store-gateways, both, or uncategorized.
  Dequeuing queries for a given tenant will rotate through the query component subqueues via simple round-robin.
  In the event that the one of the query components (ingesters or store-gateways) experience a slowdown, queries only utilizing the the other query component can continue to be serviced.
  This feature is recommended to be enabled.
  The following CLI flags must be set to true in order to be in effect:

  • -query-frontend.additional-query-queue-dimensions-enabled on the query-frontend.
  • -query-scheduler.additional-query-queue-dimensions-enabled on the query-scheduler.

• Owned series tracking in ingesters can be enabled via the -ingester.track-ingester-owned-series CLI flag.
  When enabled, inge...

2.11.0

This release contains 532 PRs from 55 authors, including new contributors Benjamin, Dominik Kepinski, Jonathan Donzallaz, Juraj Michálek, Kai.Ke, Ludovic Terrier, Luke, Maciej Lech, Matthew Penner, Michael Potter, Mihai Țimbota-Belin, Rasmus Werner Salling, Ying WANG, chencs, fayzal-g, kalle (jag), sarthaktyagi-505, whoami. Thank you!

Grafana Mimir version 2.11.0 release notes

Grafana Labs is excited to announce version 2.11 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

• Sampled logging of errors in the ingester. A high-traffic Mimir cluster can occasionally become bogged down logging high volumes of repeated errors. You can now reduce the amount of errors outputted to logs by setting a sample rate via the -ingester.error-sample-rate CLI flag.
• Add total request size instance limit for ingesters. This limit protects the ingesters against requests that together may cause an OOM. Enable this feature by setting the -ingester.instance-limits.max-inflight-push-requests-bytes CLI flag in combination with the -ingester.limit-inflight-requests-using-grpc-method-limiter CLI flag.
• Reduce the resolution of incoming native histograms samples if the incoming sample has too many buckets compared to -validation.max-native-histogram-buckets. This is enabled by default but can be turned off by setting the -validation.reduce-native-histogram-over-max-buckets CLI flag to false.
• Improved query-scheduler performance under load. This is particularly apparent for clusters with large numbers of queriers.
• Ingester to querier chunks streaming reduces the memory utilization of queriers and reduces the likelihood of OOMs.
• Ingester query request minimization reduces the number of query requests to ingesters, improving performance and resource utilization for both ingesters and queriers.

Experimental features

Grafana Mimir 2.11 includes new features that are considered experimental and disabled by default. Please use them with caution and report any issue you encounter:

• Block specified queries on a per-tenant basis. This is configured via the blocked_queries limit. See the docs for more information.
• Store metadata when ingesting metrics via OTLP. This makes metric description and type available when ingesting metrics via OTLP. You can enable this feature by setting the CLI flag -distributor.enable-otlp-metadata-storage to true.
• Reject gRPC push requests that the ingester/distributor is unable to accept before reading them into memory. You can enable this feature by using the -ingester.limit-inflight-requests-using-grpc-method-limiter and/or the -distributor.limit-inflight-requests-using-grpc-method-limiter CLI flags for the ingester and/or the distributor, respectively.
• Customize the memcached client write and read buffer size. The buffer allocated for each memcached connection can be configured via the following CLI flags:
  • For the blocks storage:
    • -blocks-storage.bucket-store.chunks-cache.memcached.read-buffer-size-bytes
    • -blocks-storage.bucket-store.chunks-cache.memcached.write-buffer-size-bytes
    • -blocks-storage.bucket-store.index-cache.memcached.read-buffer-size-bytes
    • -blocks-storage.bucket-store.index-cache.memcached.write-buffer-size-bytes
    • -blocks-storage.bucket-store.metadata-cache.memcached.read-buffer-size-bytes
    • -blocks-storage.bucket-store.metadata-cache.memcached.write-buffer-size-bytes
  • For the query frontend:
    • -query-frontend.results-cache.memcached.read-buffer-size-bytes
    • -query-frontend.results-cache.memcached.write-buffer-size-bytes
  • For the ruler storage:
    • -ruler-storage.cache.memcached.read-buffer-size-bytes
    • -ruler-storage.cache.memcached.write-buffer-size-bytes
• Configure the number of long-living workers used to process gRPC requests. This can decrease CPU usage by reducing the number of stack allocations. Configure this feature by using the -server.grpc.num-workers CLI flag.
• Enforce a limit in bytes on the PostingsForMatchers cache used by ingesters. This limit can be configured via the -blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes and -blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes CLI flags.
• Pre-allocate the pool of workers in the distributor that are used to send push requests to ingesters. This can decrease CPU usage by reducing the number of stack allocations. You can enable this feature by using the -distributor.reusable-ingester-push-worker flag.
• Include a Retry-After header in recoverable error responses from the distributor. This can protect your Mimir cluster from clients including Prometheus that default to retrying very quickly. Enable this feature by setting the -distributor.retry-after-header.enabled CLI flag.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.11 the following behavior has changed:

• The utilization-based read path limiter now operates on Go heap size instead of RSS from the Linux proc file system.

The following configuration options had been previously deprecated and are removed in Grafana Mimir 2.11:

• The CLI flag -querier.iterators.
• The CLI flag -query.batch-iterators.
• The CLI flag -blocks-storage.bucket-store.bucket-index.enabled.
• The CLI flag -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes.
• The CLI flag -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes.
• The CLI flag -blocks-storage.bucket-store.max-chunk-pool-bytes.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.13:

• The CLI flag -log.buffered; this is now the default behavior.

The following metrics are removed:

• cortex_query_frontend_workers_enqueued_requests_total; use cortex_query_frontend_enqueue_duration_seconds_count instead.

The following configuration option defaults were changed:

• The CLI flag -blocks-storage.bucket-store.index-header.sparse-persistence-enabled now defaults to true.
• The default value for the CLI flag -blocks-storage.bucket-store.index-header.lazy-loading-concurrency was changed from 0 to 4.
• The default value for the CLI flag -blocks-storage.tsdb.series-hash-cache-max-size-bytes was changed from 1GB to 350MB.
• The default value for the CLI flag -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage was changed from 10 to 15.

Bug fixes

• Ingester: Respect context cancelation during query execution. PR 6085
• Distributor: Return 529 when ingestion rate limit is hit and the distributor.service_overload_status_code_on_rate_limit_enabled flag is active. PR 6549
• Query-scheduler: Prevent accumulation of stale querier connections. PR 6100
• Packaging: Fix preremove script preventing upgrades on RHEL based OS. PR 6067

Changelog

2.11.0

Grafana Mimir

• [CHANGE] The following deprecated configurations have been removed: #6673 #6779 #6808 #6814
  • -querier.iterators
  • -querier.batch-iterators
  • -blocks-storage.bucket-store.max-chunk-pool-bytes
  • -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes
  • -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes
  • -blocks-storage.bucket-store.bucket-index.enabled
• [CHANGE] Querier: Split worker GRPC config into separate client configs for the frontend and scheduler to allow TLS to be configured correctly when specifying the tls_server_name. The GRPC config specified under -querier.frontend-client.* will no longer apply to the scheduler client, and will need to be set explicitly under -querier.scheduler-client.*. #6445 #6573
• [CHANGE] Store-gateway: enable sparse index headers by default. Sparse index headers reduce the time to load an index header up to 90%. #6005
• [CHANGE] Store-gateway: lazy-loading concurrency limit default value is now 4. #6004
• [CHANGE] General: enabled -log.buffered by default. The -log.buffered has been deprecated and will be removed in Mimir 2.13. #6131
• [CHANGE] Ingester: changed default -blocks-storage.tsdb.series-hash-cache-max-size-bytes setting from 1GB to 350MB. The new default cache size is enough to store the hashes for all series in a ingester, assuming up to 2M in-memory series per ingester and using the default 13h retention period for local TSDB blocks in the ingesters. #6130
• [CHANGE] Query-frontend: removed cortex_query_frontend_workers_enqueued_requests_total. Use cortex_query_frontend_enqueue_duration_seconds_count instead. #6121
• [CHANGE] Ingester / querier: enable ingester to querier chunks streaming by default and mark it as stable. #6174
• [CHANGE] Ingester / querier: enable ingester query request minimisation by default and mark it as stable. #6174
• [CHANGE] Ingester: changed the default value for the experimental configuration parameter -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage from 10 to 15. #6186
• [CHANGE] Ingester: /ingester/push HTTP endpoint has been removed. This endpoint was added for testing and troubleshooting, but was never documented or used for anything. #6299
• [CHANGE] Experimental...

2.11.0-rc.0

This release contains 531 PRs from 55 authors, including new contributors Benjamin, Dominik Kepinski, Jonathan Donzallaz, Juraj Michálek, Kai.Ke, Ludovic Terrier, Luke, Maciej Lech, Matthew Penner, Michael Potter, Mihai Țimbota-Belin, Rasmus Werner Salling, Ying WANG, chencs, fayzal-g, kalle (jag), renovate[bot], sarthaktyagi-505, whoami. Thank you!

Grafana Mimir version 2.11.0-rc.0 release notes

Grafana Labs is excited to announce version 2.11 of Grafana Mimir.

The highlights that follow include the top features, enhancements, and bugfixes in this release. For the complete list of changes, see the changelog.

Features and enhancements

• Sampled logging of errors in the ingester. A high-traffic Mimir cluster can occasionally become bogged down logging high volumes of repeated errors. You can now reduce the amount of errors outputted to logs by setting a sample rate via the -ingester.error-sample-rate CLI flag.
• Add total request size instance limit for ingesters. This limit protects the ingesters against requests that together may cause an OOM. Enable this feature by setting the -ingester.instance-limits.max-inflight-push-requests-bytes CLI flag in combination with the -ingester.limit-inflight-requests-using-grpc-method-limiter CLI flag.
• Reduce the resolution of incoming native histograms samples if the incoming sample has too many buckets compared to -validation.max-native-histogram-buckets. This is enabled by default but can be turned off by setting the -validation.reduce-native-histogram-over-max-buckets CLI flag to false.
• Improved query-scheduler performance under load. This is particularly apparent for clusters with large numbers of queriers.
• Ingester to querier chunks streaming reduces the memory utilization of queriers and reduces the likelihood of OOMs.
• Ingester query request minimization reduces the number of query requests to ingesters, improving performance and resource utilization for both ingesters and queriers.

Experimental features

Grafana Mimir 2.11 includes new features that are considered experimental and disabled by default. Please use them with caution and report any issue you encounter:

• Block specified queries on a per-tenant basis. This is configured via the blocked_queries limit. See the docs for more information.
• Store metadata when ingesting metrics via OTLP. This makes metric description and type available when ingesting metrics via OTLP. You can enable this feature by setting the CLI flag -distributor.enable-otlp-metadata-storage to true.
• Reject gRPC push requests that the ingester/distributor is unable to accept before reading them into memory. You can enable this feature by using the -ingester.limit-inflight-requests-using-grpc-method-limiter and/or the -distributor.limit-inflight-requests-using-grpc-method-limiter CLI flags for the ingester and/or the distributor, respectively.
• Customize the memcached client write and read buffer size. The buffer allocated for each memcached connection can be configured via the following CLI flags:
  • For the blocks storage:
    • -blocks-storage.bucket-store.chunks-cache.memcached.read-buffer-size-bytes
    • -blocks-storage.bucket-store.chunks-cache.memcached.write-buffer-size-bytes
    • -blocks-storage.bucket-store.index-cache.memcached.read-buffer-size-bytes
    • -blocks-storage.bucket-store.index-cache.memcached.write-buffer-size-bytes
    • -blocks-storage.bucket-store.metadata-cache.memcached.read-buffer-size-bytes
    • -blocks-storage.bucket-store.metadata-cache.memcached.write-buffer-size-bytes
  • For the query frontend:
    • -query-frontend.results-cache.memcached.read-buffer-size-bytes
    • -query-frontend.results-cache.memcached.write-buffer-size-bytes
  • For the ruler storage:
    • -ruler-storage.cache.memcached.read-buffer-size-bytes
    • -ruler-storage.cache.memcached.write-buffer-size-bytes
• Configure the number of long-living workers used to process gRPC requests. This can decrease CPU usage by reducing the number of stack allocations. Configure this feature by using the -server.grpc.num-workers CLI flag.
• Enforce a limit in bytes on the PostingsForMatchers cache used by ingesters. This limit can be configured via the -blocks-storage.tsdb.head-postings-for-matchers-cache-max-bytes and -blocks-storage.tsdb.block-postings-for-matchers-cache-max-bytes CLI flags.
• Pre-allocate the pool of workers in the distributor that are used to send push requests to ingesters. This can decrease CPU usage by reducing the number of stack allocations. You can enable this feature by using the -distributor.reusable-ingester-push-worker flag.
• Include a Retry-After header in recoverable error responses from the distributor. This can protect your Mimir cluster from clients including Prometheus that default to retrying very quickly. Enable this feature by setting the -distributor.retry-after-header.enabled CLI flag.

Helm chart improvements

The Grafana Mimir and Grafana Enterprise Metrics Helm chart is now released independently. See the Grafana Mimir Helm chart documentation.

Important changes

In Grafana Mimir 2.11 the following behavior has changed:

• The utilization-based read path limiter now operates on Go heap size instead of RSS from the Linux proc file system.

The following configuration options had been previously deprecated and are removed in Grafana Mimir 2.11:

• The CLI flag -querier.iterators.
• The CLI flag -query.batch-iterators.
• The CLI flag -blocks-storage.bucket-store.bucket-index.enabled.
• The CLI flag -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes.
• The CLI flag -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes.
• The CLI flag -blocks-storage.bucket-store.max-chunk-pool-bytes.

The following configuration options are deprecated and will be removed in Grafana Mimir 2.13:

• The CLI flag -log.buffered; this is now the default behavior.

The following metrics are removed:

• cortex_query_frontend_workers_enqueued_requests_total; use cortex_query_frontend_enqueue_duration_seconds_count instead.

The following configuration option defaults were changed:

• The CLI flag -blocks-storage.bucket-store.index-header.sparse-persistence-enabled now defaults to true.
• The default value for the CLI flag -blocks-storage.bucket-store.index-header.lazy-loading-concurrency was changed from 0 to 4.
• The default value for the CLI flag -blocks-storage.tsdb.series-hash-cache-max-size-bytes was changed from 1GB to 350MB.
• The default value for the CLI flag -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage was changed from 10 to 15.

Bug fixes

• Ingester: Respect context cancelation during query execution. PR 6085
• Distributor: Return 529 when ingestion rate limit is hit and the distributor.service_overload_status_code_on_rate_limit_enabled flag is active. PR 6549
• Query-scheduler: Prevent accumulation of stale querier connections. PR 6100
• Packaging: Fix preremove script preventing upgrades on RHEL based OS. PR 6067

Changelog

2.11.0-rc.0

Grafana Mimir

• [CHANGE] The following deprecated configurations have been removed: #6673 #6779 #6808 #6814
  • -querier.iterators
  • -querier.batch-iterators
  • -blocks-storage.bucket-store.max-chunk-pool-bytes
  • -blocks-storage.bucket-store.chunk-pool-min-bucket-size-bytes
  • -blocks-storage.bucket-store.chunk-pool-max-bucket-size-bytes
  • -blocks-storage.bucket-store.bucket-index.enabled
• [CHANGE] Querier: Split worker GRPC config into separate client configs for the frontend and scheduler to allow TLS to be configured correctly when specifying the tls_server_name. The GRPC config specified under -querier.frontend-client.* will no longer apply to the scheduler client, and will need to be set explicitly under -querier.scheduler-client.*. #6445 #6573
• [CHANGE] Store-gateway: enable sparse index headers by default. Sparse index headers reduce the time to load an index header up to 90%. #6005
• [CHANGE] Store-gateway: lazy-loading concurrency limit default value is now 4. #6004
• [CHANGE] General: enabled -log.buffered by default. The -log.buffered has been deprecated and will be removed in Mimir 2.13. #6131
• [CHANGE] Ingester: changed default -blocks-storage.tsdb.series-hash-cache-max-size-bytes setting from 1GB to 350MB. The new default cache size is enough to store the hashes for all series in a ingester, assuming up to 2M in-memory series per ingester and using the default 13h retention period for local TSDB blocks in the ingesters. #6130
• [CHANGE] Query-frontend: removed cortex_query_frontend_workers_enqueued_requests_total. Use cortex_query_frontend_enqueue_duration_seconds_count instead. #6121
• [CHANGE] Ingester / querier: enable ingester to querier chunks streaming by default and mark it as stable. #6174
• [CHANGE] Ingester / querier: enable ingester query request minimisation by default and mark it as stable. #6174
• [CHANGE] Ingester: changed the default value for the experimental configuration parameter -blocks-storage.tsdb.early-head-compaction-min-estimated-series-reduction-percentage from 10 to 15. #6186
• [CHANGE] Ingester: /ingester/push HTTP endpoint has been removed. This endpoint was added for testing and troubleshooting, but was never documented or used for anything. #6299...

2.9.4

Changelog

2.9.4

Grafana Mimir

• [ENHANCEMENT] Update Docker base images from alpine:3.18.3 to alpine:3.18.5. #6895

All changes in this release: mimir-2.9.3...mimir-2.9.4

2.10.5

Changelog

2.10.5

Grafana Mimir

• [ENHANCEMENT] Update Docker base images from alpine:3.18.3 to alpine:3.18.5. #6897
• [BUGFIX] Fixed possible series matcher corruption leading to wrong series being included in query results. #6886

Documentation

• [ENHANCEMENT] Document the concept of native histograms and how to send them to Mimir, migration path. #6757
• [ENHANCEMENT] Document native histograms query and visualization. #6757

All changes in this release: mimir-2.10.4...mimir-2.10.5

2.9.3

This release contains 1 PR from 1 author. Thank you!

Changelog

2.9.3

• [BUGFIX] Update go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp to 0.44 which includes a fix for CVE-2023-45142. #6637

All changes in this release: mimir-2.9.2...mimir-2.9.3

2.10.4

This release contains 3 PRs from 1 authors. Thank you!

Changelog

2.10.4

Grafana Mimir

• [BUGFIX] Update otelhttp library to v0.44.0 as a mitigation for CVE-2023-45142. #6634

All changes in this release: mimir-2.10.3...mimir-2.10.4

2.10.3

This release contains 1 PR from 1 author. Thank you!

Changelog

2.10.3

Grafana Mimir

• [BUGFIX] Update grpc-go library to 1.57.2-dev that includes a fix for a bug introduced in 1.57.1. #6419

All changes in this release: mimir-2.10.2...mimir-2.10.3