Browse Library
Sign In Start Free Trial
Close icon Search icon
Account
Browse Library Advanced Search Sign In Start Free Trial

Add to playlist

Create a Playlist

Modal Close icon
You need to login to use this feature.
  • Deep Reinforcement Learning Hands-On
  • Toc
  • feedback
Deep Reinforcement Learning Hands-On

Deep Reinforcement Learning Hands-On

By : Maxim Lapan
4.3 (34)
close
Deep Reinforcement Learning Hands-On

Deep Reinforcement Learning Hands-On

4.3 (34)
By: Maxim Lapan

Overview of this book

Deep Reinforcement Learning Hands-On is a comprehensive guide to the very latest DL tools and their limitations. You will evaluate methods including Cross-entropy and policy gradients, before applying them to real-world environments. Take on both the Atari set of virtual games and family favorites such as Connect4. The book provides an introduction to the basics of RL, giving you the know-how to code intelligent learning agents to take on a formidable array of practical tasks. Discover how to implement Q-learning on 'grid world' environments, teach your agent to buy and trade stocks, and find out how natural language models are driving the boom in chatbots.
Table of Contents (21 chapters)
close
close
Preface
Preface
Who this book is for
What this book covers
To get the most out of this book
Get in touch
Free Chapter
1
1. What is Reinforcement Learning?
1. What is Reinforcement Learning?
Learning – supervised, unsupervised, and reinforcement
RL formalisms and relations
Markov decision processes
Summary
2
2. OpenAI Gym
2. OpenAI Gym
The anatomy of the agent
Hardware and software requirements
OpenAI Gym API
The random CartPole agent
The extra Gym functionality – wrappers and monitors
Summary
3
3. Deep Learning with PyTorch
3. Deep Learning with PyTorch
Tensors
Gradients
NN building blocks
Custom layers
Final glue – loss functions and optimizers
Monitoring with TensorBoard
Example – GAN on Atari images
Summary
4
4. The Cross-Entropy Method
4. The Cross-Entropy Method
Taxonomy of RL methods
Practical cross-entropy
Cross-entropy on CartPole
Cross-entropy on FrozenLake
Theoretical background of the cross-entropy method
Summary
5
5. Tabular Learning and the Bellman Equation
5. Tabular Learning and the Bellman Equation
Value, state, and optimality
The Bellman equation of optimality
Value of action
The value iteration method
Value iteration in practice
Q-learning for FrozenLake
Summary
6
6. Deep Q-Networks
6. Deep Q-Networks
Real-life value iteration
Tabular Q-learning
Deep Q-learning
DQN on Pong
Summary
7
7. DQN Extensions
7. DQN Extensions
The PyTorch Agent Net library
Basic DQN
N-step DQN
Double DQN
Noisy networks
Prioritized replay buffer
Dueling DQN
Categorical DQN
Combining everything
Summary
References
8
8. Stocks Trading Using RL
8. Stocks Trading Using RL
Trading
Data
Problem statements and key decisions
The trading environment
Models
Training code
Results
Things to try
Summary
9
9. Policy Gradients – An Alternative
9. Policy Gradients – An Alternative
Values and policy
The REINFORCE method
REINFORCE issues
PG on CartPole
PG on Pong
Summary
10
10. The Actor-Critic Method
10. The Actor-Critic Method
Variance reduction
CartPole variance
Actor-critic
A2C on Pong
A2C on Pong results
Tuning hyperparameters
Summary
11
11. Asynchronous Advantage Actor-Critic
11. Asynchronous Advantage Actor-Critic
Correlation and sample efficiency
Adding an extra A to A2C
Multiprocessing in Python
A3C – data parallelism
A3C – gradients parallelism
Summary
12
12. Chatbots Training with RL
12. Chatbots Training with RL
Chatbots overview
Deep NLP basics
Training of seq2seq
The chatbot example
Summary
13
13. Web Navigation
13. Web Navigation
Web navigation
OpenAI Universe
Simple clicking approach
Human demonstrations
Adding text description
Things to try
Summary
14
14. Continuous Action Space
14. Continuous Action Space
Why a continuous space?
Action space
Environments
The Actor-Critic (A2C) method
Deterministic policy gradients
Distributional policy gradients
Things to try
Summary
15
15. Trust Regions – TRPO, PPO, and ACKTR
chevron up
15. Trust Regions – TRPO, PPO, and ACKTR
Introduction
Roboschool
A2C baseline
Proximal Policy Optimization
Trust Region Policy Optimization
A2C using ACKTR
Summary
16
16. Black-Box Optimization in RL
16. Black-Box Optimization in RL
Black-box methods
Evolution strategies
ES on CartPole
ES on HalfCheetah
Genetic algorithms
GA on CartPole
GA tweaks
GA on Cheetah
Summary
References
17
17. Beyond Model-Free – Imagination
17. Beyond Model-Free – Imagination
Model-based versus model-free
Model imperfections
Imagination-augmented agent
I2A on Atari Breakout
Experiment results
Summary
References
18
18. AlphaGo Zero
18. AlphaGo Zero
Board games
The AlphaGo Zero method
Connect4 bot
Connect4 results
Summary
References
Book summary
19
Other Books You May Enjoy
Other Books You May Enjoy
Leave a review - let other readers know what you think
20
Index
Index

Proximal Policy Optimization

Historically, this method came from the OpenAI team and was proposed long after TRPO (which is from 2015), but PPO is much simpler than TRPO, so we'll start from it. The paper in which it was proposed is by John Schulman et al and called Proximal Policy Optimization Algorithms, published in 2017 (arXiv:1707.06347).

The core improvement over the classical Asynchronous Advantage Actor-Critic (A3C) method is to change the expression used to estimate the PG. Instead of the gradient of logarithm probability of the action taken, the PPO method uses a different objective: the ratio between the new and the old policy scaled by the advantages.

In math form, the old A3C objective could be written as Proximal Policy Optimization. The new objective proposed by the PPO is Proximal Policy Optimization. The reason behind changing the objective is the same as for the cross-entropy method from Chapter 4, The Cross-Entropy Method: importance sampling. However, if we just start to blindly maximize this value, it may lead to a very...

Unlock full access

Continue reading for free

A Packt free trial gives you instant online access to our library of over 7000 practical eBooks and videos, constantly updated with the latest in tech
Start a 7-day FREE trial
End of Section 5
Next Section
bookmark search playlist download
font-size

Change the font size

margin-width

Change margin width

day-mode

Change background colour

Close icon Search
Country selected
Close icon Your notes and bookmarks

Delete Bookmark

Modal Close icon
Are you sure you want to delete it?
Cancel
Yes, Delete