Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

INP (still) not showing on self hosted #3322

Closed
amenk opened this issue Sep 12, 2024 · 13 comments
Closed

INP (still) not showing on self hosted #3322

amenk opened this issue Sep 12, 2024 · 13 comments

Comments

@amenk
Copy link

amenk commented Sep 12, 2024

Self-Hosted Version

24.8.0

CPU Architecture

x64_64

Docker Version

27.1.2, build d01f264

Docker Compose Version

2.29.1

Steps to Reproduce

Follow up of

getsentry/sentry-javascript#13666 & #2970

I am hosting the test file from getsentry/sentry-javascript#13666 (comment) locally via Apache with a DSN set to my self hosted instance.

Request with INP data are showing in the network panel and get a "200 ok", but nothing shows up in sentry

{"sent_at":"2024-09-12T06:41:00.797Z","trace":{"environment":"test x","release":"1726060779","public_key":"edd109a74b1c47129cf2ebe10879e6cb","trace_id":"f60691e4e5114225af15b181b9f1a77d","sample_rate":"1","sampled":"true"}}
{"type":"span"}
{"data":{"sentry.origin":"auto.http.browser.inp","sentry.op":"ui.interaction.click","sentry.source":"custom","release":"1726060779","environment":"test amk","transaction":"/inp.html","user_agent.original":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36","sentry.exclusive_time":7560,"sentry.sample_rate":1},"description":"body > button#testButton","op":"ui.interaction.click","span_id":"89f2e465eadd8189","start_timestamp":1726123250.505,"timestamp":1726123258.065,"trace_id":"f60691e4e5114225af15b181b9f1a77d","origin":"auto.http.browser.inp","exclusive_time":7560,"measurements":{"inp":{"value":7560,"unit":"millisecond"}},"is_segment":true,"segment_id":"89f2e465eadd8189"}```

sentry.conf.py feature flags:

SENTRY_FEATURES["projects:sample-events"] = False
SENTRY_FEATURES.update(
    {
        feature: True
        for feature in (
            "organizations:discover",
            "organizations:events",
            "organizations:global-views",
            "organizations:incidents",
            "organizations:integrations-issue-basic",
            "organizations:integrations-issue-sync",
            "organizations:invite-members",
            "organizations:metric-alert-builder-aggregate",
            "organizations:sso-basic",
            "organizations:sso-rippling",
            "organizations:sso-saml2",
            "organizations:performance-view",
            "organizations:advanced-search",
            "organizations:session-replay",
            "organizations:issue-platform",
            "organizations:profiling",
            "organizations:monitors",
            "organizations:dashboards-mep",
            "organizations:mep-rollout-flag",
            "organizations:dashboards-rh-widget",
            "organizations:metrics-extraction",
            "organizations:transaction-metrics-extraction",
            "organizations:performance-vitals-inp",
            "organizations:insights-entry-points",
            "organizations:insights-initial-modules",
            "organizations:insights-addon-modules",
            "projects:custom-inbound-filters",
            "projects:data-forwarding",
            "projects:discard-groups",
            "projects:plugins",
            "projects:rate-limits",
            "projects:servicehooks",
        )
        + (
            "projects:span-metrics-extraction",
            "organizations:starfish-browser-resource-module-image-view",
            "organizations:starfish-browser-resource-module-ui",
            "organizations:starfish-browser-webvitals",
            "organizations:starfish-browser-webvitals-pageoverview-v2",
            "organizations:starfish-browser-webvitals-use-backend-scores",
            "organizations:performance-calculate-score-relay",
            "organizations:starfish-browser-webvitals-replace-fid-with-inp",
            "organizations:deprecate-fid-from-performance-score",
            "organizations:performance-database-view",
            "organizations:starfish-browser-webvitals",
            "organizations:performance-screens-view",
            "organizations:starfish-browser-resource-module-ui",
            "organizations:starfish-browser-webvitals-pageoverview-v2",
            "organizations:starfish-browser-webvitals-use-backend-scores",
            "organizations:mobile-ttid-ttfd-contribution",
            "organizations:starfish-mobile-appstart",
        )  # starfish related flags
    }
)

Expected Result

INP data is showing.

Actual Result

Image

Event ID

fafed2c4858f46ea8114f133bafa79cb

@amenk
Copy link
Author

amenk commented Sep 12, 2024

🤔
Remark: after adding the insights feature flags I just did a docker compose down && docker composer up -d. I had the idea now, that I might need to rerun something else? Maybe some data migrations are missing due to the feature flags being disabled before?

@amenk
Copy link
Author

amenk commented Sep 13, 2024

I also added debug: true to my test script as suggested here by @AbhiPrasad

  <script src="https://browser.sentry-cdn.com/8.30.0/bundle.tracing.debug.min.js" crossorigin="anonymous"></script>
...
      Sentry.init({
          dsn: '......',
          release: '1726060779',
          debug: true,

console output is (after clicking the button and switching the tab)

Sentry Logger [log]: [Tracing] Starting sampled root span
  op: ui.interaction.click
  name: body
  ID: a4b35c298293e5c2
logger.js:75 Sentry Logger [log]: [Tracing] Adding an event to span: inp
logger.js:75 Sentry Logger [log]: [Tracing] Finishing "ui.interaction.click" root span "body" with ID a4b35c298293e5c2
logger.js:75 Sentry Logger [log]: Flushing outcomes...
logger.js:75 Sentry Logger [log]: No outcomes to send

@aldy505
Copy link
Collaborator

aldy505 commented Sep 21, 2024

🤔 Remark: after adding the insights feature flags I just did a docker compose down && docker composer up -d. I had the idea now, that I might need to rerun something else? Maybe some data migrations are missing due to the feature flags being disabled before?

You'll need to re-run the ./install.sh script.

But since you're on 24.8.0, I guess you'll need to match your flags to what's on that release, see here:

SENTRY_FEATURES.update(
{
feature: True
for feature in (
"organizations:discover",
"organizations:events",
"organizations:global-views",
"organizations:incidents",
"organizations:integrations-issue-basic",
"organizations:integrations-issue-sync",
"organizations:invite-members",
"organizations:metric-alert-builder-aggregate",
"organizations:sso-basic",
"organizations:sso-rippling",
"organizations:sso-saml2",
"organizations:performance-view",
"organizations:advanced-search",
"organizations:session-replay",
"organizations:issue-platform",
"organizations:profiling",
"organizations:monitors",
"organizations:dashboards-mep",
"organizations:mep-rollout-flag",
"organizations:dashboards-rh-widget",
"organizations:metrics-extraction",
"organizations:transaction-metrics-extraction",
"projects:custom-inbound-filters",
"projects:data-forwarding",
"projects:discard-groups",
"projects:plugins",
"projects:rate-limits",
"projects:servicehooks",
)
# Starfish related flags
+ (
"organizations:deprecate-fid-from-performance-score",
"organizations:indexed-spans-extraction",
"organizations:insights-entry-points",
"organizations:insights-initial-modules",
"organizations:insights-addon-modules",
"organizations:mobile-ttid-ttfd-contribution",
"organizations:performance-calculate-score-relay",
"organizations:standalone-span-ingestion",
"organizations:starfish-browser-resource-module-image-view",
"organizations:starfish-browser-resource-module-ui",
"organizations:starfish-browser-webvitals",
"organizations:starfish-browser-webvitals-pageoverview-v2",
"organizations:starfish-browser-webvitals-replace-fid-with-inp",
"organizations:starfish-browser-webvitals-use-backend-scores",
"organizations:starfish-mobile-appstart",
"projects:span-metrics-extraction",
"projects:span-metrics-extraction-addons",
)
# User Feedback related flags
+ (
"organizations:user-feedback-ingest",
"organizations:user-feedback-replay-clip",
"organizations:user-feedback-ui",
"organizations:feedback-visible",
"organizations:feedback-ingest",
"organizations:feedback-post-process-group",
)
}
)

A few things that are missing from yours:

"projects:span-metrics-extraction-addons",
"organizations:standalone-span-ingestion",
"organizations:indexed-spans-extraction",

I'd suggest to update the flags to match the one on the 24.8.0 tag first, then re-run ./install.sh

@amenk
Copy link
Author

amenk commented Sep 21, 2024

@aldy505 thank you - I will try as soon as possible.

But

SENTRY_FEATURES.update(
{
feature: True
for feature in (
"organizations:discover",
"organizations:events",
"organizations:global-views",
"organizations:incidents",
"organizations:integrations-issue-basic",
"organizations:integrations-issue-sync",
"organizations:invite-members",
"organizations:metric-alert-builder-aggregate",
"organizations:sso-basic",
"organizations:sso-rippling",
"organizations:sso-saml2",
"organizations:performance-view",
"organizations:advanced-search",
"organizations:session-replay",
"organizations:issue-platform",
"organizations:profiling",
"organizations:monitors",
"organizations:dashboards-mep",
"organizations:mep-rollout-flag",
"organizations:dashboards-rh-widget",
"organizations:metrics-extraction",
"organizations:transaction-metrics-extraction",
"projects:custom-inbound-filters",
"projects:data-forwarding",
"projects:discard-groups",
"projects:plugins",
"projects:rate-limits",
"projects:servicehooks",
)
# Starfish related flags
+ (
"organizations:deprecate-fid-from-performance-score",
"organizations:indexed-spans-extraction",
"organizations:insights-entry-points",
"organizations:insights-initial-modules",
"organizations:insights-addon-modules",
"organizations:mobile-ttid-ttfd-contribution",
"organizations:performance-calculate-score-relay",
"organizations:standalone-span-ingestion",
"organizations:starfish-browser-resource-module-image-view",
"organizations:starfish-browser-resource-module-ui",
"organizations:starfish-browser-webvitals",
"organizations:starfish-browser-webvitals-pageoverview-v2",
"organizations:starfish-browser-webvitals-replace-fid-with-inp",
"organizations:starfish-browser-webvitals-use-backend-scores",
"organizations:starfish-mobile-appstart",
"projects:span-metrics-extraction",
"projects:span-metrics-extraction-addons",
)
# User Feedback related flags
+ (
"organizations:user-feedback-ingest",
"organizations:user-feedback-replay-clip",
"organizations:user-feedback-ui",
"organizations:feedback-visible",
"organizations:feedback-ingest",
"organizations:feedback-post-process-group",
)
}
)
does not contain all the flags which are available in a release, right? I still have to add the INP stuff after syncing with the example, I guess. For example organizations:performance-vitals-inp is not in the example. Is there a list of all feature flags that are supported in a release?

@aldy505
Copy link
Collaborator

aldy505 commented Sep 21, 2024

AFAIK, organizations:performance-vitals-inp has been removed and it's now inside organizations:insights-addon-modules and projects:span-metrics-extraction-addons. So if you're on 24.8.0 and remove the performance-vitals-inp, you would be seeing INP. On my instance, I removed that too and still seeing the INP stuff.

Is there a list of all feature flags that are supported in a release?

I usually hover over https://github.com/getsentry/sentry/blob/master/src/sentry/features/temporary.py and see the commit history of that specific file. That leads somewhere..... usually.

@amenk
Copy link
Author

amenk commented Sep 21, 2024

Still not working :-(

Flags
SENTRY_FEATURES["projects:sample-events"] = False
SENTRY_FEATURES.update(
    {
        feature: True
        for feature in (
            "organizations:discover",
            "organizations:events",
            "organizations:global-views",
            "organizations:incidents",
            "organizations:integrations-issue-basic",
            "organizations:integrations-issue-sync",
            "organizations:invite-members",
            "organizations:metric-alert-builder-aggregate",
            "organizations:sso-basic",
            "organizations:sso-rippling",
            "organizations:sso-saml2",
            "organizations:performance-view",
            "organizations:advanced-search",
            "organizations:session-replay",
            "organizations:issue-platform",
            "organizations:profiling",
            "organizations:monitors",
            "organizations:dashboards-mep",
            "organizations:mep-rollout-flag",
            "organizations:dashboards-rh-widget",
            "organizations:metrics-extraction",
            "organizations:transaction-metrics-extraction",
            "projects:custom-inbound-filters",
            "projects:data-forwarding",
            "projects:discard-groups",
            "projects:plugins",
            "projects:rate-limits",
            "projects:servicehooks",

            ## added by us
            "organizations:insights-entry-points",
            "organizations:insights-initial-modules",
            "organizations:insights-addon-modules",
        )
        # Starfish related flags
        + (
            "organizations:deprecate-fid-from-performance-score",
            "organizations:indexed-spans-extraction",
            "organizations:insights-entry-points",
            "organizations:insights-initial-modules",
            "organizations:insights-addon-modules",
            "organizations:mobile-ttid-ttfd-contribution",
            "organizations:performance-calculate-score-relay",
            "organizations:standalone-span-ingestion",
            "organizations:starfish-browser-resource-module-image-view",
            "organizations:starfish-browser-resource-module-ui",
            "organizations:starfish-browser-webvitals",
            "organizations:starfish-browser-webvitals-pageoverview-v2",
            "organizations:starfish-browser-webvitals-replace-fid-with-inp",
            "organizations:starfish-browser-webvitals-use-backend-scores",
            "organizations:starfish-mobile-appstart",
            "projects:span-metrics-extraction",
            "projects:span-metrics-extraction-addons",
        )
        # User Feedback related flags
        + (
            "organizations:user-feedback-ingest",
            "organizations:user-feedback-replay-clip",
            "organizations:user-feedback-ui",
            "organizations:feedback-visible",
            "organizations:feedback-ingest",
            "organizations:feedback-post-process-group",
        )
    }
)

I re-ran install.sh and triggered sending INP values via my inp.html again

Still: "No INP data found in this project":

Sentry screenshot

And INP seems to be sent as per network tab:

Network Tab Screenshot

@amenk
Copy link
Author

amenk commented Sep 21, 2024

Wondering if this is in any way related to this similar issue which happened on the SaaS. Unfortunately there is no information on what was fixed in the SaaS exactly.

@aldy505
Copy link
Collaborator

aldy505 commented Sep 21, 2024

INP usually take a while. You might want to come back in a few hours (or maybe a few days) after sending lots of INP data.

@amenk
Copy link
Author

amenk commented Sep 21, 2024

But why is that? Does it have to do with sampling? Because my local sample rate is 1.0

Is there additional sampling happening?

How much data is needed to calculate the value? Cause I am testing that manually currently and would have to press that button 100 times then ;-) ?

I do have it running on prod, too though (but with 0.1 sample rate and an older SDK (https://browser.sentry-cdn.com/8.7.0/bundle.tracing.min.js) - so there is still hope, something will show up after the weekend...

EDIT: And I would at least expect the INP to appear instantly in the event-line, because all the other values (LCP, FCP) appear there already.

@aldy505
Copy link
Collaborator

aldy505 commented Sep 21, 2024

But why is that? Does it have to do with sampling? Because my local sample rate is 1.0

Is there additional sampling happening?

How much data is needed to calculate the value? Cause I am testing that manually currently and would have to press that button 100 times then ;-) ?

I have no idea on this, I usually don't expect Sentry stuff to appear instantly other than reported errors and alerts.

@amenk
Copy link
Author

amenk commented Sep 21, 2024

Yeah, I also noticed some kind of delay for the page views to appear together with the LCP + FCP data (maybe ~ 1 minute), but when it appears I would expect INP to be also filled.

I guess it's still not working, because I just rechecked and the INP is still empty.

With a little bit help from ChatGPT I dug into the clickhouse database:

 # docker exec -it sentry-self-hosted-clickhouse-1 clickhouse-client

SELECT 
    project_id,
    transaction_name AS page,
    start_ts AS start_timestamp,
    finish_ts AS finish_timestamp,
    arrayJoin(measurements.key) AS measurement_key,
    arrayJoin(measurements.value) AS measurement_value
FROM 
    transactions_local
WHERE 
    transaction_name = '/inp.html'
    AND arrayJoin(measurements.key) = 'inp'
ORDER BY 
    start_ts
FORMAT Vertical;

This does not yield results. Does it for you?
When I change the WHERE statement to AND arrayJoin(measurements.key) = 'fcp' I get results.

@aldy505
Copy link
Collaborator

aldy505 commented Sep 23, 2024

Yeah, I also noticed some kind of delay for the page views to appear together with the LCP + FCP data (maybe ~ 1 minute), but when it appears I would expect INP to be also filled.

I guess it's still not working, because I just rechecked and the INP is still empty.

With a little bit help from ChatGPT I dug into the clickhouse database:

 # docker exec -it sentry-self-hosted-clickhouse-1 clickhouse-client

SELECT 
    project_id,
    transaction_name AS page,
    start_ts AS start_timestamp,
    finish_ts AS finish_timestamp,
    arrayJoin(measurements.key) AS measurement_key,
    arrayJoin(measurements.value) AS measurement_value
FROM 
    transactions_local
WHERE 
    transaction_name = '/inp.html'
    AND arrayJoin(measurements.key) = 'inp'
ORDER BY 
    start_ts
FORMAT Vertical;

This does not yield results. Does it for you? When I change the WHERE statement to AND arrayJoin(measurements.key) = 'fcp' I get results.

@amenk This is what I got (removed the transaction_name obviously):

SELECT
    project_id,
    transaction_name AS page,
    start_ts AS start_timestamp,
    finish_ts AS finish_timestamp,
    arrayJoin(measurements.key) AS measurement_key,
    arrayJoin(measurements.value) AS measurement_value
FROM transactions_local
WHERE arrayJoin(measurements.key) = 'inp'
ORDER BY start_ts ASC
FORMAT Vertical

Query id: 38d8b597-d54f-4d3f-a99e-62733d557a7b

Ok.

0 rows in set. Elapsed: 0.393 sec. Processed 1.62 million rows, 56.72 MB (4.12 million rows/s., 144.22 MB/s.)
Peak memory usage: 21.17 MiB.

I don't think that's the right query for it.

@amenk
Copy link
Author

amenk commented Sep 23, 2024

Yeah, the query is wrong - I also don't see data, now, even I have now data in the dashboard.
Yes, you read correctly - it started to work now 🥳 🥳 🥳

Image

Also I can see the element attribution well, when going to interactions:

Image

For the record, I also added the flag "organizations:performance-vitals-inp" two days ago, which I found in the temporary-flag list.

But in the end of the day, it just seems to need some good amount of data to be displayed.

My local tests with /inp.html still don't display any data, but the production values are shown, now.

It would be interesting to know, how much data is actually necessary for this to be displayed, in case anyone is testing this :-)

Anyways, thank you every much @aldy505 for your help and I will close this issue.

@amenk amenk closed this as completed Sep 23, 2024
@github-actions github-actions bot locked and limited conversation to collaborators Oct 9, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
Archived in project
Archived in project
Development

No branches or pull requests

2 participants