MorphogenRL

MorphogenRL is a comprehensive and flexible framework for Reinforcement Learning research, focused on training diverse agent morphologies in procedurally generated environments. Built upon the powerful TeachMyAgent environment, this framework provides a complete pipeline for training, evaluation, and visualization, supporting various learning algorithms including PPO, Automatic Curriculum Learning (ACL), and Multi-Agent Reinforcement Learning (MARL).

The core of this project is the interaction between agents with different bodies and dynamic environments whose terrain is generated by a Compositional Pattern Producing Network (CPPN), allowing for a rich and challenging learning landscape.

Key Features

• Centralized CLI: All functionalities are managed through a single powerful entry point, run.py.
• Modular Training Pipeline: Execute a full training sequence (e.g., PPO then MARL) from a single YAML configuration file.
• Multiple Learning Algorithms:
  • PPO: Standard single-agent training using Stable Baselines 3.
  • Automatic Curriculum Learning (ACL): Automatically adjusts environment difficulty based on agent performance.
  • Multi-Agent RL (MARL): Supports both cooperative and interactive multi-agent scenarios using Ray RLlib, with integrated hyperparameter tuning via Ray Tune.
• Procedurally Generated Environments: Utilizes a CPPN to create a wide variety of terrains for the parkour environment, ensuring agents generalize well.
• Rich Library of Agent Morphologies: A large selection of predefined agent bodies, from classic bipeds to spiders, fish, and climbers.
• Powerful Tooling: Includes a comprehensive set of commands for:
  • Training, watching, and recording agents.
  • Running a full training pipeline with automatic post-training evaluation.
  • A comprehensive environment and agent body checker.
  • An integrated test suite to ensure code integrity.

Project Architecture

The project is orchestrated by the main entry point, run.py, which dispatches tasks based on user commands. These commands call various scripts in the scripts/ directory to handle everything from training and testing to visualization. The training scripts interface with the TeachMyAgent environments, which procedurally generate terrain and instantiate the specified agent bodies.

Below is a diagram illustrating the project's workflow.

Setup and Installation

It is highly recommended to use a conda environment to manage dependencies.

1. Clone the Repository

git clone https://github.com/nam-htran/MorphogenRL.git
cd MorphogenRL

2. Create and Activate Conda Environment

# Create a new conda environment (Python 3.11 is recommended)
conda create -n morphogenrl python=3.11

# Activate the environment
conda activate morphogenrl

Important Prerequisite: SWIG Installation Before installing the Python packages, you must install SWIG, a dependency for the Box2D physics engine. The version used during development was 4.0.2.

On Linux (Ubuntu/Debian):

sudo apt-get update && sudo apt-get install swig cmake

On Windows:

1. Download SWIG for Windows from the official SWIG website (e.g., swigwin-4.0.2.zip).
2. Extract the ZIP file to a permanent location (e.g., C:\swigwin-4.0.2).
3. Add the SWIG directory to your system's PATH environment variable.
4. Verify by opening a new terminal and running swig -version.

3. Install Dependencies Once SWIG is installed, install all required Python packages from the requirements.txt file. This includes ray[rllib,tune] and all other necessary libraries.

pip install -r requirements.txt

4. Convert Pre-trained Weights (One-time setup) The CPPN for procedural terrain generation uses pre-trained weights from an older TensorFlow 1.x model. A script is provided to convert them to the PyTorch format. Run this command once:

python run.py convert_weights

This creates a .pt file in TeachMyAgent/environments/envs/PCGAgents/CPPN/weights/, allowing the CPPN to function correctly.

Usage

All functionalities are accessible through run.py. Use python run.py --help to see all available commands and python run.py <command> --help for options of a specific command.

Running the Full Pipeline

This command reads configs/main_pipeline.yaml, trains the enabled stages, and automatically watches the trained agent if configured.

python run.py pipeline

Training Individual Models

Train specific algorithms directly.

Train PPO:

python run.py train ppo --run_id my_ppo_run --total_timesteps 500000

Train ACL:

python run.py train acl --run_id my_acl_run --total_stages 100

Train MARL (with Hyperparameter Tuning): Ensure use_tune: true is set in your YAML config to enable Ray Tune.

python run.py train marl --run_id my_marl_tune_run --use-tune

Watching a Trained Model

Visualize the performance of a saved single-agent model (.zip file).

python run.py watch --model_path output/ppo/my_ppo_run/models/ppo_model_final.zip

• Add --num-episodes 5 to watch for 5 episodes.
• Add --fast-forward to run the simulation at maximum speed.

Recording a Video

Record an agent's performance to an MP4 file.

python run.py record --model_path path/to/model.zip -o recordings/my_agent.mp4

Running a Demo

Launch an environment with a random agent to inspect environments and bodies.

python run.py demo --env parkour --body spider --steps 1000

Verification and Testing

Use these commands to verify the integrity of the framework.

Check all environments and bodies: This runs a short simulation for different combinations to catch any errors.

python run.py check_envs --body classic_bipedal

Run the integrated test suite: This performs a quick series of tests for training, watching, and other core functionalities.

python run.py test_suite

Environments and Agent Bodies

Environments

• stump: A simpler environment where agents traverse a track with procedurally placed stumps.
• parkour: A complex environment with terrain and ceilings generated by a CPPN, optionally including water and climbable "creepers."

Available Agent Bodies

Specify the desired body using the --body argument in any command.

• Walkers: small_bipedal, classic_bipedal, big_quadru, spider, millipede, wheel, profile_chimpanzee, old_classic_bipedal, old_big_quadru
• Swimmers: fish
• Climbers: climbing_profile_chimpanzee, climbing_chest_profile_chimpanzee
• Amphibians: amphibious_bipedal

Acknowledgements

This project is built upon and extends the excellent procedural environments from the TeachMyAgent framework. Their work on creating dynamic, parameterized environments was foundational to this research.

License

This project also use environment from TeachMyAgent, check their LICENSE on it repo https://github.com/flowersteam/TeachMyAgent. This project is licensed under the MIT License. See the LICENSE file for details.