Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Not possible to set seed for deterministic distrib model #1173

Open
matthiaskoenig opened this issue Dec 21, 2023 · 14 comments
Open

Not possible to set seed for deterministic distrib model #1173

matthiaskoenig opened this issue Dec 21, 2023 · 14 comments

Comments

@matthiaskoenig
Copy link
Collaborator

matthiaskoenig commented Dec 21, 2023 •
edited
Loading

Hi all,

I have a distrib model I want to simulate. A parameter initial assignment is sampled via an uniform distribution.
I have to set the seed for the model to be reproducible.
Unfortunately this is not possible, i.e. the integrator settings for the deterministic simulators do not have a seed setting.
There is only

(
    'relative_tolerance',
    'absolute_tolerance',
    'stiff',
    'maximum_bdf_order',
    'maximum_adams_order',
    'maximum_num_steps',
    'maximum_time_step',
    'minimum_time_step',
    'initial_time_step',
    'multiple_steps',
    'variable_step_size',
    'max_output_rows'
)

The seed can only be set for Gillespie? But a seed is required for a deterministic model with distrib information.
How would I set the seed for roadrunner in this case?

Best Matthias
@matthiaskoenig
Copy link
Collaborator Author

I found the following in the documentation

from roadrunner import Config
Config.setValue(Config.RANDOM_SEED, 42)

But:

  1. this does not work for distrib which is clearly a bug, i.e. rerunning a model script 2 times gives different values of the random variable ! (see example below).
  2. this only sets things globally, but does not allow a model wise control. We have 10.000 roadrunner instances which all need an individual seed, because we cannot control the order of instantiation of the instances on high performance clusters. So there must be a way to set the seed on a per roadrunner.ExecutableModel instance.

Minimal model attached:

import roadrunner
import pandas as pd

model_path = "spt_random.xml"
r: roadrunner.ExecutableModel = roadrunner.RoadRunner(str(model_path))

# define subset of variables as selections
selections = [
    # time
    "time",  # [min] model time
    "protein_random",
]
r.selections = selections


from roadrunner import Config
Config.setValue(Config.RANDOM_SEED, 42)

s = r.simulate(start=0, end=10, steps=1)
df = pd.DataFrame(s, columns=s.colnames)
print(df)

Here the model
spt_random.zip

Two times running the script gives:

   time  protein_random
0   0.0        0.004562
1  10.0        0.004562
   time  protein_random
0   0.0        0.962983
1  10.0        0.962983

This is a major issue and distrib models are currently not reproducible in roadrunner.

Please let me know if there is any workaround/hack in C++. We need this urgently for a publication which is about to be submitted.

@hsauro
Copy link

hsauro commented Dec 21, 2023 via email

@matthiaskoenig
Copy link
Collaborator Author

Hi all,
we are using a lot of random components when coupling our SBML models to FEM models. Currently we can't use distrib and roadrunner for handling the random initializations because we can not reproduce the random part of the model.

image

See an example attached where the roadrunner microsimulations of the FEM geometry are randomly initialized (right pattern) or other patterns set (5 examples to the left). For debugging it would be very important to reproduce the microsimulations (via a seed) and also to reproduce the random patterns. E.g. we want to run parameter scans for the same random initializations of the roadrunner instances. Without being able to set a seed this is not possible.

It would be great if one could set a seed for a roadrunner instance/deterministic simulation which is used for sampling from distrib distributions.

Best Matthias

@hsauro
Copy link

hsauro commented Jan 31, 2024 via email

@adelhpour
Copy link
Member

@hsauro I could find this seed-related issue in the tellurium repo, is that the the one you have in mind?

@hsauro
Copy link

hsauro commented Jan 31, 2024 via email

@matthiaskoenig
Copy link
Collaborator Author

@adelhpour and @luciansmith I just went through my open issues. This is the only one which I need a solution for urgently (and cannot think of any workaround).
It does not seem to be complicated to implement. Just requires the option for a seed which is set and used for the deterministic simulation. It would be great if this could be in the 2.6.0 release. If not then not.

@luciansmith
Copy link

Thank you for checking your issues! And indeed, @adelhpour is working on this, and we hope to incorporate it into the 2.6 release.

@luciansmith
Copy link

This is now implemented in the today-released 2.6.0! Do let us know if you have any issues.

@matthiaskoenig
Copy link
Collaborator Author

Hi Lucian,
thanks. How does this work? I tried the following, but it does not give the same random value for the distrib model? How do I set the seed, so I get the same value for the random protein in 2 simulations?

import roadrunner
import pandas as pd

model_path = "spt_random.xml"
r: roadrunner.RoadRunner = roadrunner.RoadRunner(str(model_path))
m: roadrunner.ExecutableModel = r.getModel()


# define subset of variables as selections
selections = [
    # time
    "time",  # [min] model time
    "protein_random",
]
r.selections = selections
m.setRandomSeed(42)
s = r.simulate(start=0, end=10, steps=1)
df = pd.DataFrame(s, columns=s.colnames)
print(df)

r.resetToOrigin()

m.setRandomSeed(42)
s = r.simulate(start=0, end=10, steps=1)
df = pd.DataFrame(s, columns=s.colnames)
print(df)

Gives 2 different random values:

   time  protein_random
0   0.0         0.15245
1  10.0         0.15245
   time  protein_random
0   0.0        0.611653
1  10.0        0.611653

@luciansmith
Copy link

Ah--it's not the ExecutableModel, it's the roadrunner object itself:

import roadrunner
import tellurium as te
import pandas as pd

model_path = "spt_random.xml"
r: roadrunner.RoadRunner = te.loada("""
    species a
    a = normal(4, 4)
""")


r.setSeed(42)
s = r.simulate(start=0, end=10, steps=1)
df = pd.DataFrame(s, columns=s.colnames)
print(df)

r.resetToOrigin()

r.setSeed(42)
s = r.simulate(start=0, end=10, steps=1)
df = pd.DataFrame(s, columns=s.colnames)
print(df)

However, I don't think we noticed the m.setRandomSeed function (sigh) so didn't change it like the others; I would think that it should behave the same way.

@luciansmith luciansmith reopened this Mar 27, 2024
@adelhpour
Copy link
Member

We actually have noticed 'setRandomSeed' function, and it is used when the user calls the 'setSeed' function of roadrunner object with a false value for 'resetModel' flag. But I did not expect the user has direct access to setRandomSeed function. I will check how it led to this issue.

@matthiaskoenig
Copy link
Collaborator Author

Hi all,
it works with the r.setSeed function. Unfortunately this has as bug sideeffect the resetting of the selections. I.e. calling r.setSeed resets the selections of the model!

import roadrunner
import pandas as pd

model_path = "spt_random.xml"
r: roadrunner.RoadRunner = roadrunner.RoadRunner(str(model_path))
m: roadrunner.ExecutableModel = r.getModel()


# define subset of variables as selections
selections = [
    # time
    "time",  # [min] model time
    "protein_random",
]
r.selections = selections

r.setSeed(42)
# r.selections = selections
# m.setRandomSeed(42)
s = r.simulate(start=0, end=10, steps=1)
df = pd.DataFrame(s, columns=s.colnames)
print(df)

r.resetToOrigin()

r.setSeed(42)
# r.selections = selections
# m.setRandomSeed(42)
s = r.simulate(start=0, end=10, steps=1)
df = pd.DataFrame(s, columns=s.colnames)
print(df)

prints

   time
0   0.0
1  10.0
   time
0   0.0
1  10.0

but should print

   time  protein_random
0   0.0        0.013265
1  10.0        0.013265
   time  protein_random
0   0.0        0.013265
1  10.0        0.013265

I.e. one has to manually reset the selections after the seed has been set !?

@luciansmith
Copy link

Happily, that was anticipated! We assumed that normally, when you set the seed you'd want to reset the model. But if not, there's a second argument to 'setSeed' that says whether or not the model should be reset, so all you have to do is use that flag:

import roadrunner
import pandas as pd

model_path = "spt_random.xml"
r: roadrunner.RoadRunner = roadrunner.RoadRunner(str(model_path))
m: roadrunner.ExecutableModel = r.getModel()


# define subset of variables as selections
selections = [
    # time
    "time",  # [min] model time
    "protein_random",
]
r.selections = selections

r.setSeed(42, resetModel=False)
# r.selections = selections
# m.setRandomSeed(42)
s = r.simulate(start=0, end=10, steps=1)
df = pd.DataFrame(s, columns=s.colnames)
print(df)

r.resetToOrigin()

r.setSeed(42, resetModel=False)
# r.selections = selections
# m.setRandomSeed(42)
s = r.simulate(start=0, end=10, steps=1)
df = pd.DataFrame(s, columns=s.colnames)
print(df)

(or just 'r.setSeed(42, False)')

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@matthiaskoenig @hsauro @luciansmith @adelhpour