Sign In Start Free Trial

Reinforcement Learning Algorithms with Python

By : Lonza

3 (3)

Reinforcement Learning Algorithms with Python

3 (3)

By: Lonza

Overview of this book

Reinforcement Learning (RL) is a popular and promising branch of AI that involves making smarter models and agents that can automatically determine ideal behavior based on changing requirements. This book will help you master RL algorithms and understand their implementation as you build self-learning agents. Starting with an introduction to the tools, libraries, and setup needed to work in the RL environment, this book covers the building blocks of RL and delves into value-based methods, such as the application of Q-learning and SARSA algorithms. You'll learn how to use a combination of Q-learning and neural networks to solve complex problems. Furthermore, you'll study the policy gradient methods, TRPO, and PPO, to improve performance and stability, before moving on to the DDPG and TD3 deterministic algorithms. This book also covers how imitation learning techniques work and how Dagger can teach an agent to drive. You'll discover evolutionary strategies and black-box optimization techniques, and see how they can improve RL algorithms. Finally, you'll get to grips with exploration approaches, such as UCB and UCB1, and develop a meta-algorithm called ESBAS. By the end of the book, you'll have worked with key RL algorithms to overcome challenges in real-world applications, and be part of the RL research community.

Preface

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Free Chapter

Section 1: Algorithms and Environments

Section 1: Algorithms and Environments

The Landscape of Reinforcement Learning

The Landscape of Reinforcement Learning

An introduction to RL

Elements of RL

Applications of RL

Summary

Questions

Further reading

Implementing RL Cycle and OpenAI Gym

Implementing RL Cycle and OpenAI Gym

Setting up the environment

OpenAI Gym and RL cycles

Development of ML models using TensorFlow

Introducing TensorBoard

Types of RL environments

Summary

Questions

Further reading

Solving Problems with Dynamic Programming

Solving Problems with Dynamic Programming

MDP

Categorizing RL algorithms

Dynamic programming

Summary

Questions

Further reading

Section 2: Model-Free RL Algorithms

Section 2: Model-Free RL Algorithms

Q-Learning and SARSA Applications

Q-Learning and SARSA Applications

Learning without a model

TD learning

SARSA

Applying SARSA to Taxi-v2

Q-learning

Applying Q-learning to Taxi-v2

Summary

Questions

Deep Q-Network

Deep Q-Network

Deep neural networks and Q-learning

DQN

DQN applied to Pong

DQN variations

Summary

Questions

Further reading

Learning Stochastic and PG Optimization

Learning Stochastic and PG Optimization

Policy gradient methods

Understanding the REINFORCE algorithm

REINFORCE with baseline

Learning the AC algorithm

Summary

Questions

Further reading

TRPO and PPO Implementation

TRPO and PPO Implementation

Roboschool

Natural policy gradient

Trust region policy optimization

Proximal Policy Optimization

Summary

Questions

Further reading

DDPG and TD3 Applications

DDPG and TD3 Applications

Combining policy gradient optimization with Q-learning

Deep deterministic policy gradient

Twin delayed deep deterministic policy gradient (TD3)

Summary

Questions

Further reading

Section 3: Beyond Model-Free Algorithms and Improvements

Section 3: Beyond Model-Free Algorithms and Improvements

Model-Based RL

Model-Based RL

Model-based methods

Combining model-based with model-free learning

ME-TRPO applied to an inverted pendulum

Summary

Questions

Further reading

Imitation Learning with the DAgger Algorithm

Imitation Learning with the DAgger Algorithm

Technical requirements

The imitation approach

Playing Flappy Bird

Understanding the dataset aggregation algorithm

IRL

Summary

Questions

Further reading

Understanding Black-Box Optimization Algorithms

Understanding Black-Box Optimization Algorithms

Beyond RL

The core of EAs

Scalable evolution strategies

Applying scalable ES to LunarLander

Summary

Questions

Further reading

Developing the ESBAS Algorithm

Developing the ESBAS Algorithm

Exploration versus exploitation

Approaches to exploration

Epochal stochastic bandit algorithm selection

Summary

Questions

Further reading

Practical Implementation for Resolving RL Challenges

Practical Implementation for Resolving RL Challenges

Best practices of deep RL

Challenges in deep RL

Advanced techniques

RL in the real world

Future of RL and its impact on society

Summary

Questions

Further reading

Assessments

Assessments

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

3 (3)

5 star

33.3%

4 star

0

3 star

33.3%

2 star

0

1 star

33.3%

Applying Q-learning to Taxi-v2

In general, Q-learning can be used to solve the same kinds of problems that can be tackled with SARSA, and because they both come from the same family (TD learning), they generally have similar performances. Nevertheless, in some specific problems, one approach can be preferred to the other. So it's useful to also know how Q-learning is implemented.

For this reason, here we'll implement Q-learning to solve Taxi-v2, the same environment that was used for SARSA. But be aware that with just a few adaptations, it can be used with every other environment with the correct characteristics. Having the results from both Q-learning and SARSA from the same environment we'll have the opportunity to compare their performance.

To be as consistent as possible, we kept some functions unchanged from the SARSA implementation. These are as follows:

...

Search

Your notes and bookmarks