[Question] Documentation on how to add your own observations

[shafeef901]

Hi,
@Danfoa @giulioturrisi thanks for the dev efforts so far :)
I'd like to know how easily one can add their own observations that can be used in the symmetry framework. For example, heightmap, object states (manipulation), or points in the environment. I imagine many of the existing functions can be easily reused for these. For my specific use case, I have some contact locations for the end effectors that the policy must learn to achieve with a boolean contact plan. So, along with robot observations, I also need to symmetrically transform the 3D contact locations, the boolean contact plan (?) and for manipulation, also the object states.

It'd be great if you could also add a documentation on how one can go about using their own observations in the symmetry framework in an intuitive manner, without getting too much into the math :)

[Danfoa]

Hey @shafeef901

Thanks for the issue. In general we still need some time to workout implementations easy and enable easy code reuse. That said if you are in a hurry, you can see examples on how to do data augmentation of custom observable spaces of a quadruped MDP (although in mujoco) here:

See the description of a quadruped_env possible list of observations here:

https://github.com/iit-DLSLab/gym-quadruped/blob/fe29c0760a43992d9f4ce65af50d5cbd612cba70/gym_quadruped/quadruped_env.pyplain=1#L35-L99

A quadruped env can be instanciated with any of these observables as part of the MDP observable space. The MDP's action and observation space are then configured programatically here:

https://github.com/iit-DLSLab/gym-quadruped/blob/fe29c0760a43992d9f4ce65af50d5cbd612cba70/gym_quadruped/quadruped_env.py?plain=1#L213-L229

And the group representations acting on the MDP's action and observation spaces are also programatically generated in this function:

https://github.com/iit-DLSLab/gym-quadruped/blob/fe29c0760a43992d9f4ce65af50d5cbd612cba70/gym_quadruped/utils/quadruped_utils.py#L327-L427

There are some caviates that we need still to consider, for instance if isaacsym uses a different joint-space DoF's order, you need to pass the new ordered list of joint names to load_symmetric_system to obtain the correct joint-space group reps.

Generic Euclidean data-augmentation

As you probably know already, to augment geometric information you use the representations provided by morphosymm or a combination of them. To see how to augment, points, vectors, pseudo-vectors, homogenous matrices, etc, see the example here:

https://github.com/Danfoa/MorphoSymm?tab=readme-ov-file#data-augmentation

For heightmaps, we need to define an appropriate representation, but this depend on the tensor structure of the heightmap. If you give us one/two weeks, ideally we can provide a more mature stable code.

Hope this helps :).

[shafeef901]

Looking forward to it. Thanks for the info so far.
I am a bit occupied at the moment for a week, but I also intend to dig deeper into it for my case.