Default precision

[nhartman94]

Hi, I was wondering if there was a motivation to use 32b vs 64b for the pytorch backend?

I was asking because I was just comparing the backends, and I needed to change the precision to 64b for the pytorch backend to get agreement with the numpy back end.

We plan to go ahead with 64b precision for our anlaysis, but I was just generically curious what the motivation for the 32b precision for pytorch backend was.

Below is just a little screen shot of the differences in the the test statistics I was seeing with the same histograms:

[matthewfeickert]

@nhartman94 I don't remember the explicit motivation for this, but I think it was because we wanted to have the user get exactly what they requested in terms of defaults as both TensorFlow and PyTorch default to 32b precision.

>>> import tensorflow as tf
>>> tf_ones = tf.ones(10)
>>> tf_ones
<tf.Tensor: shape=(10,), dtype=float32, numpy=array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32)>
>>> tf_ones.dtype

>>> import torch
>>> torch_ones = torch.ones(10)
>>> torch_ones
tensor([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.])
>>> torch_ones.dtype
torch.float32

Though so does jax

>>> from jax import numpy as jnp
>>> jax_ones = jnp.ones(10)
>>> jax_ones
DeviceArray([1., 1., 1., 1., 1., 1., 1., 1., 1., 1.], dtype=float32)
>>> jax_ones.dtype
dtype('float32')

but we there default to 64b precision in pyhf since v0.5.0 (set in PR #910)

>>> import pyhf
>>> pyhf.set_backend("jax")
>>> pyhf.tensorlib.precision
'64b'

so that's not super consistent. I've opened up Issue #1398 for this.

[matthewfeickert]

@dguest given some offline discussions we had on the surprising differences 32b vs. 64b can have, here's an example where the only thing that is happening are internal TensorFlow operations (pyhf is simply calling to TensorFlow and not doing anything pyhf-y):

>>> import pyhf
>>> pyhf.set_backend("tensorflow", precision="32b")
>>> a = pyhf.tensorlib.astensor([-2., -1., 0., 1., 2.])
>>> t = pyhf.tensorlib.erfinv(pyhf.tensorlib.erf(a))
>>> print(t)
tf.Tensor([-2.000001   -0.99999964  0.          0.99999964  1.9999981 ], shape=(5,), dtype=float32)
>>> pyhf.set_backend("tensorflow", precision="64b")
>>> a = pyhf.tensorlib.astensor([-2., -1., 0., 1., 2.])
>>> t = pyhf.tensorlib.erfinv(pyhf.tensorlib.erf(a))
>>> print(t)
tf.Tensor([-2. -1.  0.  1.  2.], shape=(5,), dtype=float64)

So given that's just one operation, you can imagine that when you're then trying to do optimizations that this compounds quite a bit.

[dguest]

More an interesting idea than a practical suggestion, but is there some way to overload the backend so that it keeps track of all the operations that are runs between tensors and you see how often you get y = (x + 1e6) - 1e6 type operations? Just bumping up the precision is the obvious fix here, but it would be cool to try to figure how where the limited precision starts to affect results.

[matthewfeickert]

This is an interesting idea. I'm not really sure off to the top of my head how we would achieve this (@lukasheinrich or @kratsg probably know better) but we could poke at this down the road.

[kratsg]

This could be done using graphs -- and you can just visualize it. Unsure of how you'd propagate quickly to see what the end result is given the non-linear minimization that goes on.

[matthewfeickert]

With PR #1400 in @nhartman94's original question will be resolved when v0.6.2 is released as the default precision for all backends will then be 64b. We'll make sure to note this in the release notes.