feat(quota): Rate limit attachments by item count #4377
Conversation
| DataCategory::Attachment => Some(Self::Bytes), | ||
| DataCategory::Session => Some(Self::Batched), |
There was a problem hiding this comment.
An aggregate session item does not support accurate counts, but this does not matter since sessions don't produce outcomes and are only rate possibly limited by setting quota to zero.
| @@ -670,13 +678,38 @@ impl Item { | |||
| /// Returns the number used for counting towards rate limits and producing outcomes. | |||
| /// | |||
| /// For attachments, we count the number of bytes. Other items are counted as 1. | |||
| pub fn quantity(&self) -> usize { | |||
| pub fn quantities(&self, purpose: CountFor) -> SmallVec<[(DataCategory, usize); 1]> { | |||
There was a problem hiding this comment.
This function now unites the old quantity and outcome_category functions.
We might be able to absorb the index_category into this function in the future, and also replace the EnvelopeSummary with this.
| DataCategory::Monitor => &mut self.monitor_quantity, | ||
| DataCategory::Span => &mut self.span_quantity, | ||
| DataCategory::ProfileChunk => &mut self.profile_chunk_quantity, | ||
| // TODO: This catch-all return looks dangerous |
There was a problem hiding this comment.
I want to remove the add_quantity function in a future PR, that will hopefully also resolve this TODO.
| @@ -1288,6 +1288,16 @@ def make_envelope(transaction_name): | |||
| "reason": "Sampled:3000", | |||
| "source": expected_source, | |||
| }, | |||
| { | |||
| "category": 22, # attachment item | |||
There was a problem hiding this comment.
We now get outcomes for attachment items.
| attachment_item_limits.longest(), | ||
| ); | ||
| attachment_limits.merge(attachment_item_limits); | ||
| } |
There was a problem hiding this comment.
This function is getting messier with every addition. I will attempt to generalize it in a future PR.
| @@ -326,8 +330,10 @@ pub struct Enforcement { | |||
| pub event: CategoryLimit, | |||
| /// The rate limit for the indexed category of the event. | |||
| pub event_indexed: CategoryLimit, | |||
| /// The combined attachment item rate limit. | |||
| /// The combined attachment bytes rate limit. | |||
| pub attachments: CategoryLimit, | |||
There was a problem hiding this comment.
I assume this name is somewhat set in stone? Because in principle I think something like attachment_bytes would be better.
There was a problem hiding this comment.
I considered it, but I did not want to mess with the serialization / python code generation.
Attachment quotas and rate limits are currently defined in bytes, so we have no way to prevent an abusively high number of very small (or even empty) attachments.
ref: #4175