Determinism: RNG State ordering

[ptheywood]

As previously mentioned in #417, there are multiple sources of non-determinism in FLAME GPU 2.

Currently, a single set of device rng states (curand states) are created and maintained, ensuring there are enough states for all threads within a layer.

This avoids race conditions, and maintains coalesced memory access when querying and updating rng state objects.

With this setup, simulations can be deterministic (ignoring other sources of non determinism).

However, if a model has optional features which can cause the order of agents in-memory to change, then for the same seed the individual agents will encounter different RNG values.

For example, a simulation may wish to output information about key events which occur in a simulation for post processing (i.e. infection transmission).

But as agent data is expensive to access on the host, and only a small fraction of agents may have information to output, agent states have been used to reduce the cost of this functionality.
I.e. Agent function condition to move agents to an to_output state if they have relevant information, iterate in a host layer function, then return to the previous state

In this case, as state changes reorder agents in memory, they will access different RNG state objects, and simulation behaviour will be different.

---

This could be resolved by agents having their "own" rng state, which follows them as they are moved around (due to sorts, birth, death, state changes etc), but this will have an associated performance cost, and increase memory usage for some models.

It would probably be best to make this feature optional per model description (at runtime, so it can be accessed in python).

i.e. (probably with a better name)

model->setPerAgentRNG(true);