Using ns3gym in Sable-Baseline3's Vevtorized Environment

[shyampparekh]

I've been experimenting with ns3gym for an LTE scenario. In order to speed up learning, I would like to use the vectorized environment support provided in Stable-Baseline3. For this, I've tried the following lines in my training python script:

n_envs = os.cpu_count()
env = make_vec_env("ns3-v0", n_envs=n_envs, vec_env_cls=SubprocVecEnv, seed=seed)

However, I get errors with this. If you've any experience with using a vectorized environment with ns3gym, any pointers for setting it up would be very helpful.

Thanks!

[shyampparekh]

A quick update. Per the above message, I've been trying to make Stable-Baselines3's SubprovVecEnv (make_vec_env with 2 or more parallel environments) work with ns3gym. DummyVecEnv works, but SubprocVecEnv seems to be running into some conflicts between pickle (messaging method used by Python for multiprocessing) and protobuf (messaging method used by ns3gym between agent/proxy in Pyhton and env in C++). I copy below the typical error messages. Please let me know if you have any ideas/suggestions for dealing with this issue.

Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 40, in _worker
remote.send(observation)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 206, in send
self._send_bytes(_ForkingPickler.dumps(obj))
File "/usr/lib/python3.10/multiprocessing/reduction.py", line 51, in dumps
cls(buf, protocol).dump(obj)
_pickle.PickleError: can't pickle repeated message fields, convert to list first
Traceback (most recent call last):
File "/home/ubuntu/repos/ns-3-allinone/ns-3.36/contrib/opengym/examples/lte_tilts/./try-test.py", line 48, in
model.learn(total_timesteps=8)
File "/usr/local/lib/python3.10/dist-packages/stable_baselines3/dqn/dqn.py", line 265, in learn
return super().learn(
File "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/off_policy_algorithm.py", line 323, in learn
total_timesteps, callback = self._setup_learn(
File "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/off_policy_algorithm.py", line 305, in _setup_learn
return super()._setup_learn(
File "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/base_class.py", line 408, in _setup_learn
self._last_obs = self.env.reset() # pytype: disable=annotation-type-mismatch
File "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 136, in reset
obs = [remote.recv() for remote in self.remotes]
File "/usr/local/lib/python3.10/dist-packages/stable_baselines3/common/vec_env/subproc_vec_env.py", line 136, in
obs = [remote.recv() for remote in self.remotes]
File "/usr/lib/python3.10/multiprocessing/connection.py", line 250, in recv
buf = self._recv_bytes()
File "/usr/lib/python3.10/multiprocessing/connection.py", line 414, in _recv_bytes
buf = self._recv(4)
File "/usr/lib/python3.10/multiprocessing/connection.py", line 383, in _recv
raise EOFError
EOFError