Learning Invariant Representations for Reinforcement Learning without Reconstruction

[KarlXing]

Summary

Link

Learning Invariant Representations for Reinforcement Learning without Reconstruction

Author/Institution

Amy Zhang, Rowan McAllister, Roberto Calandra, Yarin Gal, Sergey Levine
UCB, FAIR, McGill, Oxford

What is this

• One-sentence
  It proposes an approach of representation learning for RL to focus on task-relevant features while ignoring task-irrelevant ones based on the idea of bisimulation.

• Full
  Appropriate representation could help RL agents to learn faster and also achieve other benefits such as improved generalization. In this work, the authors propose Deep Bisimulation for Control (DBC) to learn RL control and also representation at the same time. Bisimulation metric provides a measurement of the similarity of two states based on the reward and state transition dynamics. The idea of this work is that the l1 distance of two latent state representation should approximate the bisimulation metric of the two states. One nice thing of this approach is that since bisimulation metric will consider task-relevant information only, the constraint or regularization on latent state representation learning drives the representation to ignore task-irrelevant features. This is well demonstrated by CARLA tasks.
  Another thing to notice is that they used iterative update (policy, environment model and representation) in implementation. Sometimes engineering tricks are important to make your fancy idea really work.

Comparison with previous researches. What are the novelties/good points?

Compared with other representation learning approaches such as reconstruction-based or contrastive learning based approaches.

Key points

Bisimulation metric

How the author proved effectiveness of the proposal?

1. Use MuJuCo to show their approach leads to higher rewards or faster convergence
2. Use CARLA to show the generalization advantage of their approach

Any discussions?

It's actually related to our work Domain Adaptation In Reinforcement Learning Via Latent Unified State Representation
. Our approach could be categorized into reconstruction-based state representation learning approaches. I agree representation learning could matter a lot for RL.
Bisimulation is also interesting and have a good potential for further research. It could have more utilization in RL.
Their idea is simple but interesting. Their section 5 of proof could be a good plus. Their experiments are also valid.

What should I read next?

Learning continuous latent space models for representation learning
Scalable methods for computing state similarity in deterministic Markov decision processes.

[nagataka]

• Bisimulation is an interesting concept. Especially, "Bisimulation Metrics" which softens the concept of state partitions and can be applicable to continuous state spaces sounds quite important.
  • I will read Scalable methods for computing state similarity in deterministic Markov Decision Processes
• Not only the pixel-level disentanglement but the representation considers the downstream control tasks is the good point
• I like Figure-4 as it nicely show that their latent space push similar states into neighboring location