Experiments master plan

[jurajmajerik]

Feature request

This is a WIP issue to track requested features and their priority.

Bugs

• Enforce at least 2 funnels steps before saving a goal

Metrics

• Rewrite results calculation to use HogQL
• Multiple experiment goals
• Reusable metric sets
• Visualize credible intervals
• Provide same analysis for primary and secondary metrics
• Ensure the exposure metric always makes sense (no irrelevant aggregation types)
• "Secondary metrics exposed to experiment users only" - clarify & solve the problem here
• Retention as a metric
• HogQL query as a metric

Analysis

• Winsorization (cut off extremes of a distribution)

No-code experiments

Docs

• Explain the relationship between credible intervals and win probability [TODO: link to recent docs question] (need a statistician's advice)
• Funnel vs Trend - when to use one over the other?
• Client vs server side evaluation gotchas

Miscellaneous

• Separate Experiments pricing
• Allow editing release condition from Experiments (no switching to the Feature Flag view)
• Make sure the "explore" Insights match experiment results (exact datetimes, hide irrelevant breakdowns)

Debug info

- [ ] PostHog Cloud, Debug information: [please copy/paste from https://us.posthog.com/settings/project-details#variables]
- [ ] PostHog Hobby self-hosted with `docker compose`, version/commit: [please provide]
- [ ] PostHog self-hosted with Kubernetes (deprecated, see [`Sunsetting Kubernetes support`](https://posthog.com/blog/sunsetting-helm-support-posthog)), version/commit: [please provide]