Skip to content

Commit

Permalink
docs: update topics for Blooms (#15028)
Browse files Browse the repository at this point in the history
(cherry picked from commit 469b54e)
  • Loading branch information
JStickler committed Nov 20, 2024
1 parent 066b935 commit a9ae405
Show file tree
Hide file tree
Showing 2 changed files with 31 additions and 14 deletions.
16 changes: 10 additions & 6 deletions docs/sources/get-started/labels/structured-metadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,10 @@ You should only use structured metadata in the following situations:

- If you are ingesting data in OpenTelemetry format, using Grafana Alloy or an OpenTelemetry Collector. Structured metadata was designed to support native ingestion of OpenTelemetry data.
- If you have high cardinality metadata that should not be used as a label and does not exist in the log line. Some examples might include `process_id` or `thread_id` or Kubernetes pod names.

It is an antipattern to extract information that already exists in your log lines and put it into structured metadata.
- If you are using [Explore Logs](https://grafana.com/docs/grafana-cloud/visualizations/simplified-exploration/logs/) to visualize and explore your Loki logs.
- If you are a large-scale customer, who is ingesting more than 75TB of logs a month and are using [Bloom filters](https://grafana.com/docs/loki/<LOKI_VERSION>/operations/bloom-filters/)

We do not recommend extracting information that already exists in your log lines and putting it into structured metadata.

## Attaching structured metadata to log lines

Expand All @@ -36,10 +38,11 @@ See the [Promtail: Structured metadata stage](https://grafana.com/docs/loki/<LOK

With Loki version 1.2.0, support for structured metadata has been added to the Logstash output plugin. For more information, see [logstash](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/logstash/).

{{% admonition type="warning" %}}
Structured metadata size is taken into account while asserting ingestion rate limiting.
{{< admonition type="warning" >}}
Structured metadata size is taken into account while asserting ingestion rate limiting.
Along with that, there are separate limits on how much structured metadata can be attached per log line.
```

```yaml
# Maximum size accepted for structured metadata per log line.
# CLI flag: -limits.max-structured-metadata-size
[max_structured_metadata_size: <int> | default = 64KB]
Expand All @@ -48,7 +51,8 @@ Along with that, there are separate limits on how much structured metadata can b
# CLI flag: -limits.max-structured-metadata-entries-count
[max_structured_metadata_entries_count: <int> | default = 128]
```
{{% /admonition %}}
{{< /admonition >}}
## Querying structured metadata
Expand Down
29 changes: 21 additions & 8 deletions docs/sources/query/query_accceleration.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Query acceleration (Experimental)
title: Query acceleration
menuTitle: Query acceleration
description: Provides instructions on how to write LogQL queries to benefit from query acceleration.
weight: 900
Expand All @@ -8,11 +8,12 @@ keywords:
- query acceleration
---

# Query acceleration (Experimental)
# Query acceleration

{{% admonition type="warning" %}}
Query acceleration using blooms is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided.
{{% /admonition %}}
{{< admonition type="warning" >}}
In Loki and Grafana Enterprise Logs (GEL), Query acceleration using blooms is an [experimental feature](/docs/release-life-cycle/). Engineering and on-call support is not available. No SLA is provided.
In Grafana Cloud, Query acceleration using blooms is enabled for large-scale customers that send more than 75TB of logs a month as a [public preview](/docs/release-life-cycle/) with limited support and no SLA.
{{< /admonition >}}

If [bloom filters][] are enabled, you can write LogQL queries using [structured metadata][] to benefit from query acceleration.

Expand All @@ -26,14 +27,26 @@ If [bloom filters][] are enabled, you can write LogQL queries using [structured
Queries will be accelerated for any [label filter expression][] that satisfies _all_ of the following criteria:

* The label filter expression using **string equality**, such as `| key="value"`.
* `or` and `and` operators can be used to match multiple values, such as `| detected_level="error" or detected_level="warn"`.
* _Basic_ regular expressions are automatically simplified into a supported expression:
* `| key=~"value"` is converted to `| key="value"`.
* `| key=~"value1|value2"` is converted to `| key="value1" or key="value2"`.
* `| key=~".+"` checks for existence of `key`. `.*` is not supported.
* The label filter expression is querying for structured metadata and not a stream label.
* The label filter expression is placed before any [parser expression][], [labels format expression][], [drop labels expression][], or [keep labels expression][].

To take full advantage of query acceleration with blooms, ensure that filtering structured metadata is done before any parse expression:
To take full advantage of query acceleration with blooms, ensure that filtering structured metadata is done before any parser expression:

In the following example, the query is not accelerated because the structured metadata filter, `detected_level="error"`, is after a parser stage, `json`.

```logql
{cluster="prod"} | logfmt | json | detected_level="error"
```

In the following example, the query is accelerated because the structured metadata filter is before any parser stage.

```logql
{cluster="prod"} | logfmt | json | detected_level="error" # NOT ACCELERATED: structured metadata filter is after a parse stage
{cluster="prod"} | detected_level="error" | logfmt | json # ACCELERATED: structured metadata filter is before any parse stage
{cluster="prod"} | detected_level="error" | logfmt | json
```

[bloom filters]: https://grafana.com/docs/loki/<LOKI_VERSION>/operations/bloom-filters/
Expand Down

0 comments on commit a9ae405

Please sign in to comment.