Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Introduce policy stream mapping #15982

Open
wants to merge 35 commits into
base: main
Choose a base branch
from

Conversation

DylanGuedes
Copy link
Contributor

@DylanGuedes DylanGuedes commented Jan 28, 2025

What this PR does / why we need it:
Introduces the idea of policies to Loki, which are recognizable based on the given stream selectors.
This is an improved version of #15561 and built on top of #15875.

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide (required)
  • Documentation added
  • Tests updated
  • Title matches the required conventional commits format, see here
    • Note that Promtail is considered to be feature complete, and future development for logs collection will be in Grafana Alloy. As such, feat PRs are unlikely to be accepted unless a case can be made for the feature actually being a bug fix to existing behavior.
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/setup/upgrade/_index.md
  • If the change is deprecating or removing a configuration option, update the deprecated-config.yaml and deleted-config.yaml files respectively in the tools/deprecated-config-checker directory. Example PR

@DylanGuedes DylanGuedes changed the base branch from main to retention-as-label-when-discarding January 28, 2025 14:12
@DylanGuedes DylanGuedes force-pushed the add-stream-mapping-to-loki branch from 87272ca to 243f34e Compare January 28, 2025 16:09
@github-actions github-actions bot added the type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories label Jan 28, 2025
Copy link
Contributor

github-actions bot commented Jan 28, 2025

Base automatically changed from retention-as-label-when-discarding to main January 29, 2025 12:53
@DylanGuedes DylanGuedes marked this pull request as ready for review January 29, 2025 18:45
@DylanGuedes DylanGuedes requested a review from a team as a code owner January 29, 2025 18:45
@@ -3927,6 +3927,9 @@ otlp_config:
# CLI flag: -validation.enforced-labels
[enforced_labels: <list of strings> | default = []]

# Map of policies to stream selectors with a priority. Experimental.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a default for this setting? (even if the default is nothing is set?)
Also, this could use an example in the description, since users might not know what a mapping should look like.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, default means no mapping and no policies assigned.
But good call, I'll improve the docs of this struct.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed a docs example, wdyt?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! So glad you added the example, as that's pretty complicated.

pkg/validation/limits.go Outdated Show resolved Hide resolved
pkg/compactor/retention/expiration.go Outdated Show resolved Hide resolved
pkg/distributor/validation_metrics.go Outdated Show resolved Hide resolved
pkg/distributor/distributor.go Outdated Show resolved Hide resolved
@@ -540,3 +545,40 @@ func TestExpirationChecker_IntervalMayHaveExpiredChunks(t *testing.T) {
})
}
}

func Test_PolicyStreamMapping_PolicyFor(t *testing.T) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is great! I think we should make it a bit more complete tho:

  • Two mapping policies with the same priority: Get first one matchiong. We should log an error with insight logs and some metrics but we can do that in a followup PR (I added a task to the epic for this)
  • Regexes
  • Multiple mappings per policy

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pushed a commit addressing those scenarios.

for _, policyStream := range policyStreams {
for _, m := range policyStream.Matchers {
if !m.Matches(lbs.Get(m.Name)) {
continue Outer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is not correct. IIUC here if the labels do not match the matcher for this policyStream, we continue to the next policy instead of to the next policyStream within this policy.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good find. I fixed its logic, wdyt?

pkg/compactor/retention/expiration.go Outdated Show resolved Hide resolved
pkg/compactor/retention/expiration.go Outdated Show resolved Hide resolved
found bool
)
Outer:
for policyName, policyStreams := range policyStreamMapping {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This loop is pretty similar to the RetentionPeriodFor. Still I cannot help but to think this is a O(n*m) loop that we do for every single push request we get.

Probably not a big deal if customers configure just a few mappings similarly to stream retention.
As we discussed we may need some caching here, but I'm ok with benchmarking this to see if it's possible or even doing it in a followup PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey I pushed a commit with a benchmark that I ran today. The results are:

BenchmarkDistributor_PushWithPolicies/push_without_policies-24         	   87802	     13223 ns/op	    7468 B/op	      67 allocs/op
BenchmarkDistributor_PushWithPolicies/push_with_1_policies-24          	   82648	     13417 ns/op	    7474 B/op	      67 allocs/op
BenchmarkDistributor_PushWithPolicies/push_with_10_policies-24         	   82141	     13328 ns/op	    7478 B/op	      67 allocs/op
BenchmarkDistributor_PushWithPolicies/push_with_100_policies-24        	   79417	     14192 ns/op	    7485 B/op	      67 allocs/op
BenchmarkDistributor_PushWithPolicies/push_with_1_matchers-24          	   84045	     13316 ns/op	    7489 B/op	      67 allocs/op
BenchmarkDistributor_PushWithPolicies/push_with_10_matchers-24         	   83119	     13501 ns/op	    7495 B/op	      67 allocs/op
BenchmarkDistributor_PushWithPolicies/push_with_100_matchers-24        	   76173	     14551 ns/op	    7504 B/op	      67 allocs/op
PASS

So caching isn't looking necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
size/L type/docs Issues related to technical documentation; the Docs Squad uses this label across many repositories
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants