Hands-On Q-Learning with Python

By : Nazia Habib

2.3 (3)

Buy this Book

Hands-On Q-Learning with Python

2.3 (3)

By: Nazia Habib

Buy this Book

Overview of this book

Q-learning is a machine learning algorithm used to solve optimization problems in artificial intelligence (AI). It is one of the most popular fields of study among AI researchers. This book starts off by introducing you to reinforcement learning and Q-learning, in addition to helping you become familiar with OpenAI Gym as well as libraries such as Keras and TensorFlow. A few chapters into the book, you will gain insights into model-free Q-learning and use deep Q-networks and double deep Q-networks to solve complex problems. This book will guide you in exploring use cases such as self-driving vehicles and OpenAI Gym’s CartPole problem. You will also learn how to tune and optimize Q-networks and their hyperparameters. As you progress, you will understand the reinforcement learning approach to solving real-world problems. You will also explore how to use Q-learning and related algorithms in scientific research. Toward the end, you’ll gain insight into what’s in store for reinforcement learning. By the end of this book, you will be equipped with the skills you need to solve reinforcement learning problems using Q-learning algorithms with OpenAI Gym, Keras, and TensorFlow.

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Free Chapter

Section 1: Q-Learning: A Roadmap

Brushing Up on Reinforcement Learning Concepts

What is RL?

States, actions, and rewards

Key concepts in RL

SARSA versus Q-learning – on-policy or off?

Summary

Questions

Getting Started with the Q-Learning Algorithm

Technical requirements

Demystifying MDPs

Your Q-learning agent in its environment

Fine-tuning your model – learning, discount, and exploration rates

MABP – a classic exploration versus exploitation problem

Optimal versus safe paths – revisiting SARSA

Summary

Questions

Setting Up Your First Environment with OpenAI Gym

Technical requirements

Getting started with OpenAI Gym

Exploring the Taxi-v2 environment

Creating a baseline agent

Summary

Questions

Teaching a Smartcab to Drive Using Q-Learning

Technical requirements

Getting to know your learning agent

Implementing your agent

The learning parameters – alpha, gamma, and epsilon

Model-tuning and tracking your agent's long-term performance

Summary

Questions

Section 2: Building and Optimizing Q-Learning Agents

Building Q-Networks with TensorFlow

Technical requirements

A brief overview of neural networks

Taking a closer look

Implementing a neural network with NumPy

Neural networks and Q-learning

Building your first Q-network

Summary

Questions

Further reading

Digging Deeper into Deep Q-Networks with Keras and TensorFlow

Technical requirements

Introducing CartPole-v1

Getting started with the CartPole task

Building a DQN to solve the CartPole problem

Testing and results

Adding in experience replay

Building further on DQNs

Summary

Questions

Further reading

Section 3: Advanced Q-Learning Challenges with Keras, TensorFlow, and OpenAI Gym

Decoupling Exploration and Exploitation in Multi-Armed Bandits

Technical requirements

Probability distributions and ongoing knowledge

Revisiting a simple bandit problem

Multi-armed bandit strategy overview

Contextual bandits and state diagrams

Thompson sampling and the Bayesian control rule

Solving a multi-armed bandit problem in Python – user advertisement clicks

Multi-armed bandits in experimental design

Summary

Questions

Further reading

Further Q-Learning Research and Future Projects

Google's DeepMind and the future of Q-learning

OpenAI Gym and RL research

More OpenAI Gym environments

Contextual bandits and probability distributions

Summary

Questions

Further reading

Assessments

Chapter 1, Brushing Up on Reinforcement Learning Concepts

Chapter 2, Getting Started with the Q-Learning Algorithm

Chapter 3, Setting Up Your First Environment with OpenAI Gym

Chapter 4, Teaching a Smartcab to Drive Using Q-Learning

Chapter 5, Building Q-Networks with TensorFlow

Chapter 6, Digging Deeper into Deep Q-Networks with Keras and TensorFlow

Chapter 7, Decoupling Exploration and Exploitation in Multi-Armed Bandits

Chapter 8, Further Q-Learning Research and Future Projects

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

2.3 (3)

5 star

33.3%

4 star

3 star

2 star

1 star

66.7%

Implementing a neural network with NumPy

In this section, we are implementing a fully-connected ReLU classifier using NumPy.

Note that, in practice, we wouldn't implement a simple neural network with this level of detail; this is only for demonstration purposes so that we can get comfortable with the matrix multiplication and feedforward structure that is involved.

As mentioned in the previous section, NumPy has no internal structure for handling gradients or computation graphs; it is a broadly-used framework within Python for scientific computing. However, we can apply matrix operations to NumPy objects to simulate a two-layer network that incorporates feedforward and backpropagation.

All of the code for this section can be found in the GitHub repository for this chapter; note that not all of the code is published here.

We begin by importing the required package, as follows...