I need help to make the PPO learn how to walk

[AndreaRossetto]

Hi,
I'm traying to make a biped robot learn to walk in IsaacLab. In the training I use 5000 env and I can make the models stand for a while... The maximal and mean timesteps are high but the min timesteps are not going up, same for the rewards.

Last attempt:

Previous attempt:

As you can see the previus attempt the PPO at ~80k steps start to getting worst

seed: 42

# Models are instantiated using skrl's model instantiator utility
models:
  separate: False
  policy:  # see gaussian_model parameters
    class: GaussianMixin
    clip_actions: False
    clip_log_std: True
    min_log_std: -20.0
    max_log_std: 2.0
    initial_log_std: 0.0
    network:
      - name: net
        input: OBSERVATIONS #["robot-state"]
        layers:
          - linear: [512]
          - linear: [256]
          - linear: [128]
        activations: ["elu", "elu", "elu"]
    output: ACTIONS
  value:  # see deterministic_model parameters
    class: DeterministicMixin
    clip_actions: False
    network:
      - name: net
        input: OBSERVATIONS #["robot-state"]
        layers:
          - linear: [512]
          - linear: [256]
          - linear: [128]
        activations: ["elu", "elu", "elu"]
    output: ONE


# Rollout memory
memory:
        class: RandomMemory
        memory_size: -1  # automatically determined (same as agent:rollouts)


# A rollout is the process of running the agent in the environment to collect data. In many cases,
#   it's a complete simulation of the agent's behavior from the start until termination. Also called episode in some contexts.

agent:
  class:                              PPO
  rollouts:                           24                   # horizon_length
  learning_epochs:                    5
  mini_batches:                       4                    # horizon_length * num_actors / minibatch_size
  discount_factor:                    0.995                # gamma
  lambda:                             0.95
  learning_rate:                      1.0e-03
  learning_rate_scheduler:            KLAdaptiveLR
  learning_rate_scheduler_kwargs:
    kl_threshold: 0.025
  state_preprocessor: null
  state_preprocessor_kwargs: null
  value_preprocessor: null
  value_preprocessor_kwargs: null
  random_timesteps: 0
  learning_starts: 0
  grad_norm_clip: 1.0
  ratio_clip: 0.2
  value_clip: 0.2
  clip_predicted_values: True
  entropy_loss_scale: 0.001
  value_loss_scale: 1.0
  kl_threshold: 0.0
  rewards_shaper_scale: 1.0
  time_limit_bootstrap: False

  # logging and checkpoint
  experiment:
    directory: "uDroid"
    experiment_name: ""
    write_interval: 800 # TensorBoard writing interval (timesteps)
    checkpoint_interval: 800 # interval for checkpoints (timesteps)
    store_separately: False


# Sequential trainer
trainer:
  class: SequentialTrainer
  timesteps: 1000000 #720000 # rollouts * max_epochs
  environment_info: log

track_lin_vel_xy_exp = RewTerm(
      func=mdp.track_lin_vel_xy_exp, weight=1.5, params={"command_name": "base_velocity", "std": math.sqrt(1.0)}
  )
  track_ang_vel_z_exp = RewTerm(
      func=mdp.track_ang_vel_z_exp, weight=0.75, params={"command_name": "base_velocity", "std": math.sqrt(0.5)}
  )

  # Large reward for maintaining an upright posture
  upright = RewTerm(func=mdp.upright_posture_bonus, 
                    weight=2.0, 
                    params={
                              "asset_cfg": SceneEntityCfg("robot", body_names="Full_base_link"),
                              "threshold": 0.85
                          }
                  )

  feet_air_time = RewTerm(
      func=mdp.feet_air_time_positive_biped,
      weight = 0.5,
      params = {
          "sensor_cfg": SceneEntityCfg("feet_contact_sensor", body_names="foot_joint1_.*"),
          "command_name": "base_velocity",
          "threshold_min": 0.4,
          "threshold_max": 0.8,
      },
  )

  # Large penalty for termination
  termination_penalty = RewTerm(func=mdp.is_terminated, weight=-200.0)

  feet_slide = RewTerm(
      func=mdp.feet_slide,
      weight = -0.25,
      params = {
          "sensor_cfg": SceneEntityCfg("feet_contact_sensor", body_names="foot_joint1_.*"),
          "asset_cfg": SceneEntityCfg("robot", body_names="foot_joint1_.*"),
      },
  )

  # -- Penalties to Discourage Unnecessary Movement --
  # Penalize linear velocity of the torso
  torso_velocity_penalty = RewTerm(func=mdp.base_lin_vel_l2, weight=-1.5)
  # Penalize angular velocity of the torso (wobbling)
  torso_wobble_penalty = RewTerm(func=mdp.base_ang_vel_l2, weight=-0.05)

  # -- Secondary Goal: Encourage a Neutral Resting Pose --
  # Penalize deviation from the default joint positions to encourage stability    
  joint_deviation_hip = RewTerm(
      func=mdp.joint_deviation_l1,
      weight=-0.5,
      params={"asset_cfg": SceneEntityCfg("robot", joint_names=[".*"])},
  )

  undesired_contacts = RewTerm(
      func=mdp.undesired_contacts,
      weight=-0.5,
      params={"sensor_cfg": SceneEntityCfg("contact_forces", body_names=["Hip.*", "HAA.*", "Thigh.*", "KAF.*"]), "threshold": 10.0},
  )
  
  # -- penalties
  pen_joint_vel_l2 = RewTerm(func=mdp.joint_vel_l2, weight=-5.0e-05)
  pen_joint_accel = RewTerm(func=mdp.joint_acc_l2, weight=-1.25e-07)
  joint_torques_l2 = RewTerm(func=mdp.joint_torques_l2, weight=-1.0e-5)
  # Penalize large changes in action between steps to encourage smooth movements
  action_rate_l2 = RewTerm(func=mdp.action_rate_l2, weight=-0.01)
  # Penalize large action magnitudes to conserve energy
  action_magnitude_penalty = RewTerm(func=mdp.action_l2, weight=-0.1)

What is wrong ? the parameters? I have to increase the number of parallel models? the rewards are not correct?