You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
As @glentner points out, the first thing an ipython/jupyter user is going to do is:
import cupynumeric as np
a = np.ones((100_000, 100_000))
b = np.ones((100_000, 100_000))
c = a @ b
c[:10,:10] # print out some elements
Currently this would cause the entire global array to be inline-mapped (gathered) onto each node's system memory, before it gets printed out, possibly causing OOM along the way.
If we were to inline-map subsets of the full array, then we would run the risk of having multiple overlapping read-write views over the same data, whose coherence we would have to manage explicitly.
A saving grace for this case is that printing (i.e. calling __str__) only requires a (temporary) read-only view over the (sub-)array, so we could (a) do a one-shot inline mapping of a sub-region (requires a Legate extension), (b) wrap the PhysicalStore (i.e. the memory buffer returned by the inline mapping) with a NumPy array, (c) print out the NumPy array, (d) immediately unmap. We should only do this in the deferred case (in the eager case the array is already fully in python memory), and only if we're not printing the full array (no reason to do a copy if we're gonna need to inline-map the full array anyway).
Some caveats:
We don't actually know for sure this was the issue @glentner ran into in his experiments, would like to confirm that e.g. printing out c.max() works, and also what, if any, the error output was for the original c[:10,:10].
This won't do anything if the user just tries to print the whole array, so we should avoid pulling the whole array in __str__ if it's too big -- selectively pull (through a series of partial one-shot mappings) a subset of elements and only print those out, similar to what NumPy does when printing out a large array.
This won't do anything if the user (indirectly) asks to inline-map the whole array for a reason other than printing, e.g. to pass to h5py. We probably want to have some warning in this case, and suggest to the user to use some alternative API that doesn't need to pull the entire array into one address space. This API doesn't always exist, but it does for (some) HDF5 functionality.
The text was updated successfully, but these errors were encountered:
As @glentner points out, the first thing an ipython/jupyter user is going to do is:
Currently this would cause the entire global array to be inline-mapped (gathered) onto each node's system memory, before it gets printed out, possibly causing OOM along the way.
If we were to inline-map subsets of the full array, then we would run the risk of having multiple overlapping read-write views over the same data, whose coherence we would have to manage explicitly.
A saving grace for this case is that printing (i.e. calling
__str__
) only requires a (temporary) read-only view over the (sub-)array, so we could (a) do a one-shot inline mapping of a sub-region (requires a Legate extension), (b) wrap the PhysicalStore (i.e. the memory buffer returned by the inline mapping) with a NumPy array, (c) print out the NumPy array, (d) immediately unmap. We should only do this in the deferred case (in the eager case the array is already fully in python memory), and only if we're not printing the full array (no reason to do a copy if we're gonna need to inline-map the full array anyway).Some caveats:
c.max()
works, and also what, if any, the error output was for the originalc[:10,:10]
.__str__
if it's too big -- selectively pull (through a series of partial one-shot mappings) a subset of elements and only print those out, similar to what NumPy does when printing out a large array.h5py
. We probably want to have some warning in this case, and suggest to the user to use some alternative API that doesn't need to pull the entire array into one address space. This API doesn't always exist, but it does for (some) HDF5 functionality.The text was updated successfully, but these errors were encountered: