[Investigation] Smart Limits for Detection Rules

[Mikaayenson]

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

With #3842 we limited the number of versions per rule released down to 2 versions. We've been asked by D&R and our PM @approksiu to investigate the LOE to implement smart limits elastic/kibana#187645 . There are hurdles to implement this feature for both publishing and ingesting rules. This issue tracker will help to understand any technical limitations on our end.

This issue is purely an investigation to provide more insight to what limitations /options we can control from this side.

Desired Solution

Details on options / hurdles to implement smart limits. Examples:

• Limit based on latest over a period of time?
• Limit based on rule changes?
• etc.

Considered Alternatives

We may not need to implement complicated smart limits at all based on some initial testing by @xcrzx who may have a solution that supports the number of rule versions published to a limit that we theoretically would not reach (considering the qualitative balance between the number of rules vs maintaining and publishing a number of high quality rules).

This would be another option if they are successful.

Additional Context

• [Security Solution] Smart limits for the package with prebuilt rules kibana#187645
• Since we rely on epr for historical context, we will need to traverse history and at least ship one time or maintain someway a list of the baseline (initial) rule so that the rule customization feature coming upstream will know how to best diff and provide a better out of box experience for users. We need to consider that some rules may have been released at a stack version (with a rule version) that we no longer release to. We also need to consider other things like rule forks and rule releases against different stack versions.

[Mikaayenson]

Update Oct 24

@xcrzx Was able to test loading rules with a limit of about 20k rules. We may be able to remove the limit altogether starting in 8.17. We still need to wait on the serverless testing to confirm.

Long term, we all agree it would be better to have a distribution mechanism that uses git (where kibana pulls directly from the detection-rules repo branches).

[xcrzx]

Hey @Mikaayenson, I finished testing the package limits on Serverless last week, and the results are in this PR. Long story short, there were no issues installing a 15k-rule package on a Serverless project. Memory consumption was stable and well within the 1GB limit, averaging around 40-45%, which is a great result. I couldn’t test a 20k-rule package due to some other test environment limitations, but I don’t anticipate any significant differences.

We can set the total assets-per-package limit pretty high, like the tested 15k. We’ll probably never reach it, but it’s safer to keep the limit in place in case we get close before migrating from Fleet to a Git-based rule distribution system. Do you know how many saved objects would be in the package if we include current rules with all their historical versions?

[shashank-elastic]

Hey @xcrzx today we have roughly around ~1250 rules in each rule packages for versions such as 8.12 --- 8.15 ( ball park numbers that can vary on actual) refer.

Previous to the limiting of historical rules to the latest 2 we would have roughly around ~3700 rules in a package for older releases refer

Without having the latest rules added to the EPR historical data, its difficult to arrive at the actual numbers of rules that would be in the packages once we remove the limiting of versions. Given that the tested limit was roughly around 15k, I don't see the rule assets reaching this number immediately (may be grow over time as we add more versions to the package) even if we build today without limiting historical versions.

[xcrzx]

Thanks for the info, @shashank-elastic!

The stream-based package installation has been merged, so we can plan the limit increase. It’s set for release in Kibana 8.17, meaning we can raise the package limit starting with that version. For earlier Kibana versions, we should still limit the package to the two most recent versions per rule to prevent OOM issues

Let’s also discuss what we are planning to do if a package reaches the 15k limit. I’ve been considering an algorithm with these steps:

1. Select all current rule versions and add them to the package.
2. Gather all historical rule versions and sort them from newest to oldest.
3. Add the newest historical versions until the package size reaches 15k.

This would allow us to discard only the oldest rule versions, the ones users are least likely to need, providing an optimal rule upgrade experience.

However, I recall there were challenges with retrieving the creation date of rule versions, which could complicate sorting from newest to oldest. Is that still the case?

I’d also welcome any thoughts on this approach or alternative implementation ideas. Let me know if you see a different way forward.

cc @Mikaayenson @banderror