Fix usage of enableCameraSensors

[tylerlum]

Goal

Enable proper use of the enableCameraSensors config parameter.

Problem

The current behavior results in enableCameraSensors always being False.

Reason

enable_camera_sensors = config.get("enableCameraSensors", False) always returns False because this is reading from the wrong part of the config. In almost all of the task yaml files, enableCameraSensors is defined under task.env, but in the code it is looking under task. The use of the get and default value made this bug pass by silently, but I noticed it when trying to get my camera sensors to work in a headless training environment.

I also fixed the AllegroHands' yaml files to be consistent with the others (before they were under task.task).

How To Check

Can manually check using:

git grep -B 200 enableCameraSensors:

I also wrote a small python script called check_for_enableCameraSensors.py that does a similar check if placed in IsaacGymEnvs/isaacgymenvs.

# IsaacGymEnvs/isaacgymenvs/check_for_enableCameraSensors.py
from hydra import compose, initialize
from omegaconf import OmegaConf
import isaacgym  # Need to import isaacgym before pytorch
from isaacgymenvs.tasks import isaacgym_task_map
from typing import Dict, List, Optional, Any


def find_path_to_keyword(
    dictionary: Dict[str, Any], keyword: str, path: Optional[List[str]] = None
) -> List[str]:
    """
    EXAMPLE INPUTS:
    --------------
    data = {
        "task": {"env": 1},
        "hello": 2
    }
    keyword = "env"

    EXAMPLE OUTPUTS:
    --------------
    ["task", "env"]
    """

    if path is None:
        path = []

    for key, value in dictionary.items():
        path.append(key)

        if key == keyword:
            return path

        if isinstance(value, dict):
            nested_path = find_path_to_keyword(value, keyword, path)
            if nested_path:
                return nested_path

        path.pop()

    return []


initialize(config_path="./cfg")

isaacgym_tasks = list(isaacgym_task_map.keys())
for task in isaacgym_tasks:
    # Read task yaml
    try:
        cfg = compose(config_name="config", overrides=[f"task={task}"])
    except:
        print(f"Failed to load {task}")
        continue

    # Get path to enableCameraSensors
    path_to_keyword = find_path_to_keyword(
        dictionary=OmegaConf.to_container(cfg), keyword="enableCameraSensors"
    )
    if len(path_to_keyword) == 0:
        print(f"enableCameraSensors not in {task}")
        continue
    
    print(f"Path to enableCameraSensors in {task}: {path_to_keyword}")

cd IsaacGymEnvs/isaacgymenvs
python check_for_enableCameraSensors.py

Before Change

Note the ['task, 'task', 'enableCameraSensors'] in AllegroHand is inconsistent (it was the same in AllegroHandManualDR and AllegroHandADR too, but failed to load for some reason).

Path to enableCameraSensors in AllegroHand: ['task', 'task', 'enableCameraSensors']
enableCameraSensors not in AllegroKuka
Failed to load AllegroKukaTwoArms
Failed to load AllegroHandManualDR
Failed to load AllegroHandADR
Path to enableCameraSensors in Ant: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in Anymal: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in AnymalTerrain: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in BallBalance: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in Cartpole: ['task', 'env', 'enableCameraSensors']
enableCameraSensors not in FactoryTaskGears
enableCameraSensors not in FactoryTaskInsertion
enableCameraSensors not in FactoryTaskNutBoltPick
enableCameraSensors not in FactoryTaskNutBoltPlace
enableCameraSensors not in FactoryTaskNutBoltScrew
Path to enableCameraSensors in FrankaCabinet: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in FrankaCubeStack: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in Humanoid: ['task', 'env', 'enableCameraSensors']
enableCameraSensors not in HumanoidAMP
Path to enableCameraSensors in Ingenuity: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in Quadcopter: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in ShadowHand: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in Trifinger: ['task', 'env', 'enableCameraSensors']

After

Note the consistent ['task, 'env', 'enableCameraSensors'] now.

Path to enableCameraSensors in AllegroHand: ['task', 'env', 'enableCameraSensors']
enableCameraSensors not in AllegroKuka
Failed to load AllegroKukaTwoArms
Failed to load AllegroHandManualDR
Failed to load AllegroHandADR
Path to enableCameraSensors in Ant: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in Anymal: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in AnymalTerrain: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in BallBalance: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in Cartpole: ['task', 'env', 'enableCameraSensors']
enableCameraSensors not in FactoryTaskGears
enableCameraSensors not in FactoryTaskInsertion
enableCameraSensors not in FactoryTaskNutBoltPick
enableCameraSensors not in FactoryTaskNutBoltPlace
enableCameraSensors not in FactoryTaskNutBoltScrew
Path to enableCameraSensors in FrankaCabinet: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in FrankaCubeStack: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in Humanoid: ['task', 'env', 'enableCameraSensors']
enableCameraSensors not in HumanoidAMP
Path to enableCameraSensors in Ingenuity: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in Quadcopter: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in ShadowHand: ['task', 'env', 'enableCameraSensors']
Path to enableCameraSensors in Trifinger: ['task', 'env', 'enableCameraSensors']

[tylerlum]

Found related forum post: https://forums.developer.nvidia.com/t/error-in-creating-camera-sensor-handle-while-headless/239808/2