feat(events tracking): add abstract class and logging implementation #80117
Conversation
github-actions
bot
added
the
Scope: Backend
src/sentry/utils/event_tracker.py
Outdated
| snuba_topic_put | ||
| billing_topic_put |
There was a problem hiding this comment.
updated the list
| START = "start" | ||
| END = "end" |
There was a problem hiding this comment.
Consider making the enum an integer. That would take a lot less memory in redis.
The galaxy brain version would be to create a bitmap. One bit per phase. Each event only needs 12 bits to encode all phases. https://redis.io/docs/latest/develop/data-types/bitmaps/
There was a problem hiding this comment.
not writing to redis with the logging implementation but i get the idea
src/sentry/utils/event_tracker.py
Outdated
|
|
||
| def __init__(self): | ||
| self.logger = logging.getLogger("EventTracker") | ||
| logging.basicConfig(level=logging.INFO) |
There was a problem hiding this comment.
doesn't this modify the root logger for all of the application?
There was a problem hiding this comment.
removed
src/sentry/utils/event_tracker.py
Outdated
| raise NotImplementedError | ||
|
|
||
|
|
||
| class EventTracker(EventTrackerBackend): |
There was a problem hiding this comment.
Do you need the base class for any reason? Are you planning to provide a second implementation? I'd just remove it if not and make it a bit simpler
There was a problem hiding this comment.
Are you planning to provide a second implementation?
not anytime soon, for the purpose of tracking event stages logger based implementation should be sufficient. Maybe another metric based implementation can be used for refunding?
There was a problem hiding this comment.
im gonna delete for now
| self.logger = logging.getLogger("EventTracker") | ||
| logging.basicConfig(level=logging.INFO) | ||
|
|
||
| def record_event_stage_status(self, event_id: str, status: EventStageStatus): |
There was a problem hiding this comment.
can you please add sampling logic in here? we'll be in trouble if we log every event. i think it's safer to not require the caller to remember to do that and build sampling as a core concept into this class instead
There was a problem hiding this comment.
added, default is 0.01
There was a problem hiding this comment.
you don't want random though do you? don't you need consistent sampling so that the same events get sampled at every stage of the pipeline?
victoria-yining-huang
force-pushed
the
vic/add_logging_module
branch
from
a147a53
to
c09a7c4
Compare
victoria-yining-huang
force-pushed
the
vic/add_logging_module
branch
from
2ebf8d5
to
8b5234b
Compare
| """ | ||
| Records how far an event has made it through the ingestion pipeline. | ||
| """ | ||
| self.logger.info(f"EventTracker recorded event {event_id} - {status.value}") |
There was a problem hiding this comment.
if you are building log analysis tools on top of this structured logging might make life easier
victoria-yining-huang
force-pushed
the
vic/add_logging_module
branch
from
9d4217c
to
3143ea2
Compare
Codecov ReportAll modified and coverable lines are covered by tests ✅ ✅ All tests successful. No failed tests found. Additional details and impacted files@@ Coverage Diff @@
## master #80117 +/- ##
========================================
Coverage 78.12% 78.12%
========================================
Files 7163 7166 +3
Lines 316700 316849 +149
Branches 43663 43685 +22
========================================
+ Hits 247414 247536 +122
- Misses 62944 62961 +17
- Partials 6342 6352 +10 |
github-actions
bot
added
the
Scope: Frontend
|
🚨 Warning: This pull request contains Frontend and Backend changes! It's discouraged to make changes to Sentry's Frontend and Backend in a single pull request. The Frontend and Backend are not atomically deployed. If the changes are interdependent of each other, they must be separated into two pull requests and be made forward or backwards compatible, such that the Backend or Frontend can be safely deployed independently. Have questions? Please ask in the |
Bundle ReportChanges will increase total bundle size by 20.69kB (0.06%) ⬆️. This is within the configured threshold ✅ Detailed changes
|
victoria-yining-huang
force-pushed
the
vic/add_logging_module
branch
from
af42f0e
to
3143ea2
Compare
design doc
need to track the completion of each stage, to 1) compute events conversion rates 2) enable debugging visibility into where events are being dropped
the usage will be heavily sampled to not blow up traffic