Streaming blocks onto GPU for problems larger than GPU memory

[Dizzixx]

First I would like to say I am really impressed with Taichi and how effortlessly it extends C and CUDA like performance to python users, amazing stuff!

I am working on a physics based (ie - not just pretty pictures) MPM code and am currently evaluating my options which are to use some kind of existing research code, write something myself, or use existing libraries to minimize the effort involved in writing something myself. Ultimately my problems are small to medium in size (by MPM standards) and compute efficiency is critical (ie - want to run many 1000's to 10's of 1000's of simulations on a timescale of hours, days, a week) rending is convenient for the purpose of V&V and to distill results to a readily communicable format but I am really just trying to drive the physics through time to an end state and then extract directly some or multiple discrete quantities.

I have played around with the Taichi simulation examples and am very impressed with the out of the box performance but have a number of questions regarding extending the framework to:

• Use alternative/more performant implementations (such as [1][2][3] etc.)
• Handle problems larger than the available GPU memory while still utilizing GPU speedup (I played around with the mpm128 and at somewhere between 1-2 million particles with only 512^2 grid points and I run out of memory on my laptop with 8GB VRAM, same issue for euler example with grid between 1024^2 and 2056^2)
• Extend effortlessly to multiple GPU's on larger systems including where they are non-homogeneous
• Include physics based equations of state and constitutive models
• Use multiple grids and or domain partitioning to achieve mixed solver solutions (ie - Euler fluid + MPM solids or Navier-Stokes fluid + MPM solids, FEM+MPM, PFEM fluid+MPM solid, etc)
• Handle non-uniform and AMR grids

I fully appreciate that I am unlikely to achieve the same performance as built into Taichi attempting to roll my own, but would need to achieve a number of the above points with Taichi for it to even be a viable solution path.

I imagine others have dabbled with similar questions at one point or another (I believe the authors of multiple of the awesome examples might have some input) and am hopeful the community might be able to provide feedback as to whether some of these things are readily achievable (for instance the EOS and constitutive models which should be fairly straight forward to implement), might be in the pipeline or otherwise being worked on by others (AMR? mulit-solver coupling?), or are simply asking too much (multiple grids, streaming solutions onto/off of GPU where memory limited? etc.)

I appreciate any feedback you can provide and look forward to your replies.