You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Use ChangesOfVariables and InverseFunctions (#212)
* Add ChangesOfVariables and InverseFunctions to deps
* Replace forward by with_logabsdet_jacobian
* Replace Base.inv with InverseFunctions.inverse
* Improve deprecation scheme for forward
Co-authored-by: David Widmann <[email protected]>
* Improve deprecation scheme for inv
* Test forward and inv deprecations
* Apply suggestions from code review
Co-authored-by: David Widmann <[email protected]>
* Fixes regarding with_logabsdet_jacobian and inverse
* Fix with_logabsdet_jacobian for NamedComposition
* Fix deprecation of inv
* Use inverse instead of inv for Composed
* Use with_logabsdet_jacobian instead of forward
* Workaround for intermittent failures in Dirichlet test
* Use with_logabsdet_jacobian instead of forward
* Use with_logabsdet_jacobian instead of forward
* Add rrules for combine with PartitionMask
Zygote-generated pullback for `combine(m::PartitionMask, x_1, x_2, x_3)`
fails with `no method matching zero(::Type{Nothing})`.
* Use inv instead of inverse for numbers
* Apply suggestions from code review
Co-authored-by: David Widmann <[email protected]>
* Whitespace fix.
Co-authored-by: David Widmann <[email protected]>
* Move combine rrule and add test
* Apply suggestions from code review
Co-authored-by: David Widmann <[email protected]>
* Use @test_deprecated
Co-authored-by: David Widmann <[email protected]>
* Use @test_deprecated
Co-authored-by: David Widmann <[email protected]>
* Use inverse instead of inv
* Use test_inverse and test_with_logabsdet_jacobian
* Use inverse instead of inv
* Increase version number to v0.9.12
* Reexport with_logabsdet_jacobian and inverse
* Increase package version to v0.10.0
Co-authored-by: David Widmann <[email protected]>
Pretty neat, huh? `Inverse{Logit}` is also a `Bijector` where we've defined `(ib::Inverse{<:Logit})(y)` as the inverse transformation of `(b::Logit)(x)`. Note that it's not always the case that `inv(b) isa Inverse`, e.g. the inverse of `Exp` is simply `Log` so `inv(Exp()) isa Log` is true.
136
+
Pretty neat, huh? `Inverse{Logit}` is also a `Bijector` where we've defined `(ib::Inverse{<:Logit})(y)` as the inverse transformation of `(b::Logit)(x)`. Note that it's not always the case that `inverse(b) isa Inverse`, e.g. the inverse of `Exp` is simply `Log` so `inverse(Exp()) isa Log` is true.
137
137
138
138
#### Dimensionality
139
139
One more thing. See the `0` in `Inverse{Logit{Float64}, 0}`? It represents the *dimensionality* of the bijector, in the same sense as for an `AbstractArray` with the exception of `0` which means it expects 0-dim input and output, i.e. `<:Real`. This can also be accessed through `dimension(b)`:
In the computation of both `logpdf` and `logpdf_forward` we need to compute `log(abs(det(jacobian(inv(b), y))))` and `log(abs(det(jacobian(b, x))))`, respectively. This computation is available using the `logabsdetjac` method
204
+
In the computation of both `logpdf` and `logpdf_forward` we need to compute `log(abs(det(jacobian(inverse(b), y))))` and `log(abs(det(jacobian(b, x))))`, respectively. This computation is available using the `logabsdetjac` method
which is always the case for a differentiable bijection with differentiable inverse. Therefore if you want to compute `logabsdetjac(b⁻¹, y)` and we know that `logabsdetjac(b, b⁻¹(y))` is actually more efficient, we'll return `-logabsdetjac(b, b⁻¹(y))` instead. For some bijectors it might be easy to compute, say, the forward pass `b(x)`, but expensive to compute `b⁻¹(y)`. Because of this you might want to avoid doing anything "backwards", i.e. using `b⁻¹`. This is where `forward` comes to good use:
221
+
which is always the case for a differentiable bijection with differentiable inverse. Therefore if you want to compute `logabsdetjac(b⁻¹, y)` and we know that `logabsdetjac(b, b⁻¹(y))` is actually more efficient, we'll return `-logabsdetjac(b, b⁻¹(y))` instead. For some bijectors it might be easy to compute, say, the forward pass `b(x)`, but expensive to compute `b⁻¹(y)`. Because of this you might want to avoid doing anything "backwards", i.e. using `b⁻¹`. This is where `with_logabsdet_jacobian` comes to good use:
In fact, the purpose of `forward` is to just _do the right thing_, not necessarily "forward". In this function we'll have access to both the original value `x` and the transformed value `y`, so we can compute `logabsdetjac(b, x)` in either direction. Furthermore, in a lot of cases we can re-use a lot of the computation from `b(x)` in the computation of `logabsdetjac(b, x)`, or vice-versa. `forward(b, x)` will take advantage of such opportunities (if implemented).
235
+
In fact, the purpose of `with_logabsdet_jacobian` is to just _do the right thing_, not necessarily "forward". In this function we'll have access to both the original value `x` and the transformed value `y`, so we can compute `logabsdetjac(b, x)` in either direction. Furthermore, in a lot of cases we can re-use a lot of the computation from `b(x)` in the computation of `logabsdetjac(b, x)`, or vice-versa. `with_logabsdet_jacobian(b, x)` will take advantage of such opportunities (if implemented).
236
236
237
237
#### Sampling from `TransformedDistribution`
238
238
At this point we've only shown that we can replicate the existing functionality. But we said `TransformedDistribution isa Distribution`, so we also have `rand`:
@@ -241,7 +241,7 @@ At this point we've only shown that we can replicate the existing functionality.
241
241
julia> y =rand(td) # ∈ ℝ
242
242
0.999166054552483
243
243
244
-
julia> x =inv(td.transform)(y) # transform back to interval [0, 1]
244
+
julia> x =inverse(td.transform)(y) # transform back to interval [0, 1]
julia> td =transformed(Normal(), b⁻¹) # x ∼ 𝓝(0, 1) then b(x) ∈ (0, 1)
@@ -280,7 +280,7 @@ It's worth noting that `support(Beta)` is the _closed_ interval `[0, 1]`, while
280
280
```julia
281
281
td =transformed(Beta())
282
282
283
-
inv(td.transform)(rand(td))
283
+
inverse(td.transform)(rand(td))
284
284
```
285
285
286
286
will never result in `0` or `1` though any sample arbitrarily close to either `0` or `1` is possible. _Disclaimer: numerical accuracy is limited, so you might still see `0` and `1` if you're lucky._
@@ -335,7 +335,7 @@ julia> # Construct the transform
335
335
bs =bijector.(dists) # constrained-to-unconstrained bijectors for dists
Another useful function is the `forward(d::Distribution)` method. It is similar to `forward(b::Bijector)` in the sense that it does a forward pass of the entire process "sample then transform" and returns all the most useful quantities in process using the most efficent computation path.
484
+
Another useful function is the `forward(d::Distribution)` method. It is similar to `with_logabsdet_jacobian(b::Bijector, x)` in the sense that it does a forward pass of the entire process "sample then transform" and returns all the most useful quantities in process using the most efficent computation path.
485
485
486
486
```julia
487
487
julia> x, y, logjac, logpdf_y =forward(flow) # sample + transform and returns all the useful quantities in one pass
@@ -542,41 +542,43 @@ Logit{Float64}(0.0, 1.0)
542
542
julia>b(0.6)
543
543
0.4054651081081642
544
544
545
-
julia>inv(b)(y)
545
+
julia>inverse(b)(y)
546
546
Tracked 2-element Array{Float64,1}:
547
547
0.3078149833748082
548
548
0.72380041667891
549
549
550
550
julia>logabsdetjac(b, 0.6)
551
551
1.4271163556401458
552
552
553
-
julia>logabsdetjac(inv(b), y) # defaults to `- logabsdetjac(b, inv(b)(x))`
553
+
julia>logabsdetjac(inverse(b), y) # defaults to `- logabsdetjac(b, inverse(b)(x))`
554
554
Tracked 2-element Array{Float64,1}:
555
555
-1.546158373866469
556
556
-1.6098711387913573
557
557
558
-
julia>forward(b, 0.6) # defaults to `(rv=b(x), logabsdetjac=logabsdetjac(b, x))`
A Bijector representing composition of bijectors. composel and composer results in a Composed for which application occurs from left-to-right and right-to-left, respectively.
667
669
668
-
Note that all the alternative ways of constructing a Composed returns a Tuple of bijectors. This ensures type-stability of implementations of all relating methdos, e.g. inv.
670
+
Note that all the alternative ways of constructing a Composed returns a Tuple of bijectors. This ensures type-stability of implementations of all relating methods, e.g. inverse.
669
671
670
672
If you want to use an Array as the container instead you can do
671
673
@@ -713,9 +715,9 @@ The distribution interface consists of:
713
715
#### Methods
714
716
The following methods are implemented by all subtypes of `Bijector`, this also includes bijectors such as `Composed`.
715
717
- `(b::Bijector)(x)`: implements the transform of the `Bijector`
716
-
- `inv(b::Bijector)`: returns the inverse of `b`, i.e. `ib::Bijector` s.t. `(ib ∘ b)(x) ≈ x`. In most cases this is `Inverse{<:Bijector}`.
718
+
- `inverse(b::Bijector)`: returns the inverse of `b`, i.e. `ib::Bijector` s.t. `(ib ∘ b)(x) ≈ x`. In most cases this is `Inverse{<:Bijector}`.
- `forward(b::Bijector, x)`: returns named tuple `(rv=b(x), logabsdetjac=logabsdetjac(b, x))` in the most efficient manner.
720
+
- `with_logabsdet_jacobian(b::Bijector, x)`: returns the tuple `(b(x), logabsdetjac(b, x))` in the most efficient manner.
719
721
- `∘`, `composel`, `composer`: convenient and type-safe constructors for `Composed`. `composel(bs...)` composes s.t. the resulting composition is evaluated left-to-right, while `composer(bs...)` is evaluated right-to-left. `∘` is right-to-left, as excepted from standard mathematical notation.
720
722
- `jacobian(b::Bijector, x)` [OPTIONAL]: returns the Jacobian of the transformation. In some cases the analytical Jacobian has been implemented for efficiency.
721
723
- `dimension(b::Bijector)`: returns the dimensionality of `b`.
@@ -725,7 +727,7 @@ For `TransformedDistribution`, together with default implementations for `Distri
725
727
- `bijector(d::Distribution)`: returns the default constrained-to-unconstrained bijector for `d`
726
728
- `transformed(d::Distribution)`, `transformed(d::Distribution, b::Bijector)`: constructs a `TransformedDistribution` from `d` and `b`.
727
729
- `logpdf_forward(d::Distribution, x)`, `logpdf_forward(d::Distribution, x, logjac)`: computes the `logpdf(td, td.transform(x))` using the forward pass, which is potentially faster depending on the transform at hand.
728
-
- `forward(d::Distribution)`: returns `(x = rand(dist), y = b(x), logabsdetjac = logabsdetjac(b, x), logpdf = logpdf_forward(td, x))` where `b = td.transform`. This combines sampling from base distribution and transforming into one function. The intention is that this entire process should be performed in the most efficient manner, e.g. the `logabsdetjac(b, x)` call might instead be implemented as `- logabsdetjac(inv(b), b(x))` depending on which is most efficient.
730
+
- `forward(d::Distribution)`: returns `(x = rand(dist), y = b(x), logabsdetjac = logabsdetjac(b, x), logpdf = logpdf_forward(td, x))` where `b = td.transform`. This combines sampling from base distribution and transforming into one function. The intention is that this entire process should be performed in the most efficient manner, e.g. the `logabsdetjac(b, x)` call might instead be implemented as `- logabsdetjac(inverse(b), b(x))` depending on which is most efficient.
729
731
730
732
# Bibliography
731
733
1. Rezende, D. J., & Mohamed, S. (2015). Variational Inference With Normalizing Flows. [arXiv:1505.05770](https://arxiv.org/abs/1505.05770v6).
0 commit comments