You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The PSIS paper notes it is sometimes useful to compute h-specific (function/expectation-specific) diagnostics, which more specifically diagnose the quality of importance sampling for the expectation in question. From discussions with @avehtari, this can be done with the following procedure. Given importance ratios r(θ) and function evaluations h(θ), we:
Compute w(θ) as r(θ) with a Pareto-smoothed upper tail.
Compute v(θ) = h(θ) r(θ). Fit the GPD to upper tails of both v(θ) and -v(θ) to get kₕᵤ and kₕₗ. Compute kₕ=max(kₕᵤ, kₕₗ).
Warn for either k > 0.7 or kₕ > 0.7
Return w(θ), k, and kₕ. The user would then estimate 𝔼[h] as sum(h .* w) ./ sum(w).
Likewise, we can return h-specific ESS estimates by simply replacing w in the ESS computation with v.
A few things to consider:
Providing this requires decoupling diagnosing from smoothing. i.e. for some uses one might just want to fit the GPD and check the diagnostic without modifying any weights. (Edit: see Allow diagnosing without smoothing #26 26)
PSISResult currently can't accommodate 3 smoothing results. Perhaps it should be extended, or a different object should be returned if h is provided.
Currently the API assumes users sample from the proposal distribution once and then smooth for many different target distributions simultaneously. While this is what is needed for LOO, I suspect that diagnosing for many different h's simultaneously could be more useful. Perhaps the API should support both.
RE (2), since Monte Carlo is just a special case of IS where all weights are uniform, the same function could be used to diagnose all univariate marginals of a posterior sample to identify problems with estimating means and variances of parameters. This should be supported.
The PSIS paper notes it is sometimes useful to compute
h
-specific (function/expectation-specific) diagnostics, which more specifically diagnose the quality of importance sampling for the expectation in question. From discussions with @avehtari, this can be done with the following procedure. Given importance ratiosr(θ)
and function evaluationsh(θ)
, we:w(θ)
asr(θ)
with a Pareto-smoothed upper tail.v(θ) = h(θ) r(θ)
. Fit the GPD to upper tails of bothv(θ)
and-v(θ)
to getkₕᵤ
andkₕₗ
. Computekₕ=max(kₕᵤ, kₕₗ)
.k > 0.7
orkₕ > 0.7
w(θ)
,k
, andkₕ
. The user would then estimate𝔼[h]
assum(h .* w) ./ sum(w)
.Likewise, we can return
h
-specific ESS estimates by simply replacingw
in the ESS computation withv
.A few things to consider:
PSISResult
currently can't accommodate 3 smoothing results. Perhaps it should be extended, or a different object should be returned ifh
is provided.h
's simultaneously could be more useful. Perhaps the API should support both.v
. Between this feature and that, some changes may need to be made to the internals.The text was updated successfully, but these errors were encountered: