MVP: Cost attribution #10269
MVP: Cost attribution #10269
Conversation
ying-jeanne
force-pushed
the
final-cost-attribution
branch
from
5165a5b to
6f36b5f
Compare
ying-jeanne
force-pushed
the
final-cost-attribution
branch
from
6f36b5f to
077a94a
Compare
ying-jeanne
force-pushed
the
final-cost-attribution
branch
from
077a94a to
f04c28f
Compare
| @@ -502,6 +525,18 @@ func (s *seriesStripe) remove(ref storage.SeriesRef) { | |||
| } | |||
|
|
|||
| s.active-- | |||
| if s.cat != nil { | |||
| if idx == nil { | |||
There was a problem hiding this comment.
Same here, we should assume this isn't nil. Just skipping the removal will break the numbers forever.
There was a problem hiding this comment.
vendor and update in commit 4706bde
There was a problem hiding this comment.
Please update the active series tracker tests with the costattribution.Tracker, otherwise the new code isn't tested.
pkg/ingester/ingester.go
Outdated
| idx, err := db.Head().Index() | ||
| if err != nil { | ||
| level.Warn(i.logger).Log("msg", "failed to get the index of the TSDB head", "user", userID, "err", err) | ||
| idx = nil | ||
| } |
There was a problem hiding this comment.
As commented previously, we should never proceed without an index.
If you check the implementation of db.Head().Index() it never returns an error. We have three options here:
- Skip tenants if they don't have index: this is the least effort one.
- Panic if err is not nil, this is ugly
- Update mimir-prometheus to add a
MustIndex() IndexReadermethod that does not return an error, and use that one.
There was a problem hiding this comment.
PR in mimir-prometheus grafana/mimir-prometheus#811
There was a problem hiding this comment.
vendor and update in commit 4706bde
| if t.Cfg.CostAttributionRegistryPath != "" { | ||
| reg := prometheus.NewRegistry() | ||
| var err error | ||
| t.CostAttributionManager, err = costattribution.NewManager(3*time.Minute, time.Minute, t.Cfg.CostAttributionEvictionInterval, util_log.Logger, t.Overrides, reg) |
There was a problem hiding this comment.
I think these values should not be hardcoded.
pkg/costattribution/tracker.go
Outdated
| } | ||
| } | ||
|
|
||
| func (t *Tracker) shouldDelete(deadline int64) bool { |
There was a problem hiding this comment.
I would rename this method to "recovered from overflow". The fact that it should be deleted is a decision of the Manager.
There was a problem hiding this comment.
addressed in 1ab1f00
pkg/costattribution/tracker.go
Outdated
| } | ||
|
|
||
| func (t *Tracker) shouldDelete(deadline int64) bool { | ||
| if t.cooldownUntil != nil && t.cooldownUntil.Load() < deadline { |
There was a problem hiding this comment.
You can't check whether t.cooldownUntil without holding the mutex here.
There was a problem hiding this comment.
addressed in fe8a1e5
pkg/costattribution/manager.go
Outdated
| newTrackedLabels := m.limits.CostAttributionLabels(userID) | ||
|
|
||
| // sort the labels to ensure the order is consistent | ||
| sort.Slice(newTrackedLabels, func(i, j int) bool { |
There was a problem hiding this comment.
Is it safe to modify the slice returned by m.limits.CostAttributionLabels(userID)?
There was a problem hiding this comment.
modify the copy instead, addressed in 888d8b0
What this PR does
This is the follow up of #9733,
The PR intent to export extra attributed metrics in distributor and ingester, in order to get sample received, sample discarded and active_series attributed by cost attribution label.
Which issue(s) this PR fixes or relates to
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.