Deep Q-learning with TensorFlow

Keywords: Python, TensorFlow, Deep Reinforcement Learning, Deep Q-learning, Deep Q Networks

Table of contents

1. Installation
2. Introduction
   2.1 Goal
   2.2 Results
3. Project structure
4. Environments
5. Project usage
   5.1 Train a DQN agent to play SpaceInvaders via Reinforcement Learning
   5.2 Test the training DQN agent to play SpaceInvaders via Reinforcement Learning
6. Todo
7. Resources

1. Installation

This project was designed for:

• Python 3.6
• TensorFlow 1.12.0

Please install requirements & project:

$ cd /path/to/project/
$ git clone https://github.com/filippogiruzzi/deep_q_learning.git
$ cd deep_q_learning/
$ pip3 install -r requirements.txt
$ pip3 install -e . --user --upgrade

2. Introduction

2.1 Goal

The purpose of this project is to design and implement Deep Q-learning algorithms such as Deep Q Networks (DQN) or Double Deep Q Networks (DDQN) to play games (e.g Atari games) with Reinforcement Learning.

The global outline for a DQN algorithm is the following:

2.2 Results

This section shows qualitative results for the initial random DQN agent (on the left), and the DQN agent during training, after approximately 5000 episodes (on the right).

3. Project structure

The project deep_q_learning/ has the following structure:

• dqn/models/: pre-processing, Q-value & target network estimators, DQN agent
• dqn/training/: online training
• dqn/inference/: AI playing

4. Environments

The project supports the OpenAI Gym only for now. More games shall be added in the future.

5. Project usage

$ cd /path/to/project/deep_q_learning/dqn/

5.1 Train a DQN agent to play SpaceInvaders via Reinforcement Learning

$ python3 training/train.py

5.2 Test the training DQN agent to play SpaceInvaders via Reinforcement Learning

$ python3 inference/test.py

6. Todo

• Finish remodeling
• Full training on Google colab
• Add inference scripts
• Add recordings
• Add quantitative results
• Add DDQN model

7. Resources

This project was widely inspired by:

• Reinforcement Learning tutorials, Denny Britz, Github
• Reinforcement Learning tutorials, Woongwon, Youngmoo, Hyeokreal, Uiryeong, Keon, Github
• CS 285 (UC Berkeley) - Deep Reinforcement Learning, Sergey Levine, Course page
• Personal Reinforcement Learning notes, Filippo Giruzzi, Github
• Playing Atari with Deep Reinforcement Learning, Mnih et al., 2013, Arxiv
• Deep Reinforcement Learning to play Space Invaders, Deshai, Banerjee, Project report