Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add metrics and generic metrics backend #2355

Merged
merged 3 commits into from
Aug 30, 2023
Merged

Add metrics and generic metrics backend #2355

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
37fbf58
add metrics backend
hubertdeng123 Aug 18, 2023
1207c8a
Merge branch 'master' into hubertdeng123/add-metrics
hubertdeng123 Aug 29, 2023
403e929
uncomment platform in dc file
hubertdeng123 Aug 30, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -278,6 +278,18 @@ services:
snuba-issue-occurrence-consumer:
<<: *snuba_defaults
command: consumer --storage search_issues --consumer-group generic_events_group --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
snuba-metrics-consumer:
<<: *snuba_defaults
command: consumer --storage metrics_raw --consumer-group snuba-metrics-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
snuba-generic-metrics-distributions-consumer:
<<: *snuba_defaults
command: consumer --storage generic_metrics_distributions_raw --consumer-group snuba-gen-metrics-distributions-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
snuba-generic-metrics-sets-consumer:
<<: *snuba_defaults
command: consumer --storage generic_metrics_sets_raw --consumer-group snuba-gen-metrics-sets-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
snuba-generic-metrics-counters-consumer:
<<: *snuba_defaults
command: consumer --storage generic_metrics_counters_raw --consumer-group snuba-gen-metrics-counters-consumers --auto-offset-reset=latest --max-batch-time-ms 750 --no-strict-offset-reset
snuba-replacer:
<<: *snuba_defaults
command: replacer --storage errors --auto-offset-reset=latest --no-strict-offset-reset
Expand All @@ -290,6 +302,9 @@ services:
snuba-subscription-consumer-transactions:
<<: *snuba_defaults
command: subscriptions-scheduler-executor --dataset transactions --entity transactions --auto-offset-reset=latest --no-strict-offset-reset --consumer-group=snuba-transactions-subscriptions-consumers --followed-consumer-group=transactions_group --delay-seconds=60 --schedule-ttl=60 --stale-threshold-seconds=900
snuba-subscription-consumer-metrics:
<<: *snuba_defaults
command: subscriptions-scheduler-executor --dataset metrics --entity metrics_sets --entity metrics_counters --auto-offset-reset=latest --no-strict-offset-reset --consumer-group=snuba-metrics-subscriptions-consumers --followed-consumer-group=snuba-metrics-consumers --delay-seconds=60 --schedule-ttl=60 --stale-threshold-seconds=900
snuba-profiling-profiles-consumer:
<<: *snuba_defaults
command: consumer --storage profiles --auto-offset-reset=latest --max-batch-time-ms 1000 --no-strict-offset-reset
Expand Down Expand Up @@ -345,6 +360,15 @@ services:
transactions-consumer:
<<: *sentry_defaults
command: run consumer ingest-transactions --consumer-group ingest-consumer
metrics-consumer:
<<: *sentry_defaults
command: run consumer ingest-metrics --consumer-group metrics-consumer
generic-metrics-consumer:
<<: *sentry_defaults
command: run consumer ingest-generic-metrics --consumer-group generic-metrics-consumer
billing-metrics-consumer:
<<: *sentry_defaults
command: run consumer billing-metrics-consumer --consumer-group billing-metrics-consumer
ingest-replay-recordings:
<<: *sentry_defaults
command: run consumer ingest-replay-recordings --consumer-group ingest-replay-recordings
Expand All @@ -369,6 +393,12 @@ services:
subscription-consumer-transactions:
<<: *sentry_defaults
command: run consumer transactions-subscription-results --consumer-group query-subscription-consumer
subscription-consumer-metrics:
<<: *sentry_defaults
command: run consumer metrics-subscription-results --consumer-group query-subscription-consumer
subscription-consumer-generic-metrics:
<<: *sentry_defaults
command: run consumer generic-metrics-subscription-results --consumer-group query-subscription-consumer
sentry-cleanup:
<<: *sentry_defaults
image: sentry-cleanup-self-hosted-local
Expand Down
2 changes: 1 addition & 1 deletion install/create-kafka-topics.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ echo "${_group}Creating additional Kafka topics ..."
# NOTE: This step relies on `kafka` being available from the previous `snuba-api bootstrap` step
# XXX(BYK): We cannot use auto.create.topics as Confluence and Apache hates it now (and makes it very hard to enable)
EXISTING_KAFKA_TOPICS=$($dcr -T kafka kafka-topics --list --bootstrap-server kafka:9092 2>/dev/null)
NEEDED_KAFKA_TOPICS="ingest-attachments ingest-transactions ingest-events ingest-replay-recordings profiles ingest-occurrences"
NEEDED_KAFKA_TOPICS="ingest-attachments ingest-transactions ingest-events ingest-replay-recordings profiles ingest-occurrences ingest-metrics ingest-performance-metrics"
for topic in $NEEDED_KAFKA_TOPICS; do
if ! echo "$EXISTING_KAFKA_TOPICS" | grep -qE "(^| )$topic( |$)"; then
$dcr kafka kafka-topics --create --topic $topic --bootstrap-server kafka:9092
Expand Down
12 changes: 12 additions & 0 deletions sentry/sentry.conf.example.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -188,6 +188,13 @@ def get_internal_network():

SENTRY_DIGESTS = "sentry.digests.backends.redis.RedisBackend"

###################
# Metrics Backend #
###################

SENTRY_RELEASE_HEALTH = "sentry.release_health.metrics.MetricsReleaseHealthBackend"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this live in https://github.com/getsentry/snuba/blob/master/snuba/settings/settings_self_hosted.py instead?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No this is the right place for the setting. This lets sentry know to load sessions metric data from the metrics dataset (which is the change you're trying to make).

Snuba will serve up sessions or metrics and isn't aware of what the product is doing really

SENTRY_RELEASE_MONITOR = "sentry.release_health.release_monitor.metrics.MetricReleaseMonitorBackend"

##############
# Web Server #
##############
Expand Down Expand Up @@ -271,6 +278,11 @@ def get_internal_network():
"organizations:session-replay",
"organizations:issue-platform",
"organizations:profiling",
"organizations:dashboards-mep",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are these orgs designed to test the new snuba containers somehow?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

those are feature flags that are set at the org level, and utilize the new snuba containers

"organizations:mep-rollout-flag",
"organizations:dashboards-rh-widget",
"organizations:metrics-extraction",
"organizations:transaction-metrics-extraction",
"projects:custom-inbound-filters",
"projects:data-forwarding",
"projects:discard-groups",
Expand Down