Sign In Start Free Trial

Hands-On Reinforcement Learning for Games

By : Micheal Lanham

5 (3)

Hands-On Reinforcement Learning for Games

5 (3)

By: Micheal Lanham

Overview of this book

With the increased presence of AI in the gaming industry, developers are challenged to create highly responsive and adaptive games by integrating artificial intelligence into their projects. This book is your guide to learning how various reinforcement learning techniques and algorithms play an important role in game development with Python. Starting with the basics, this book will help you build a strong foundation in reinforcement learning for game development. Each chapter will assist you in implementing different reinforcement learning techniques, such as Markov decision processes (MDPs), Q-learning, actor-critic methods, SARSA, and deterministic policy gradient algorithms, to build logical self-learning agents. Learning these techniques will enhance your game development skills and add a variety of features to improve your game agent’s productivity. As you advance, you’ll understand how deep reinforcement learning (DRL) techniques can be used to devise strategies to help agents learn from their actions and build engaging games. By the end of this book, you’ll be ready to apply reinforcement learning techniques to build a variety of projects and contribute to open source applications.

Preface

Preface

Who this book is for

What this book covers

To get the most out of this book

Get in touch

Section 1: Exploring the Environment

Section 1: Exploring the Environment

Free Chapter

Understanding Rewards-Based Learning

Understanding Rewards-Based Learning

Technical requirements

Understanding rewards-based learning

Introducing the Markov decision process

Using value learning with multi-armed bandits

Exploring Q-learning with contextual bandits

Summary

Questions

Dynamic Programming and the Bellman Equation

Dynamic Programming and the Bellman Equation

Introducing DP

Understanding the Bellman equation

Building policy iteration

Building value iteration

Playing with policy versus value iteration

Exercises

Summary

Monte Carlo Methods

Monte Carlo Methods

Understanding model-based and model-free learning

Introducing the Monte Carlo method

Adding RL

Playing the FrozenLake game

Using prediction and control

Exercises

Summary

Temporal Difference Learning

Temporal Difference Learning

Understanding the TCA problem

Introducing TDL

Applying TDL to Q-learning

Exploring TD(0) in Q-learning

Running off- versus on-policy

Exercises

Summary

Exploring SARSA

Exploring SARSA

Exploring SARSA on-policy learning

Using continuous spaces with SARSA

Extending continuous spaces

Working with TD (λ) and eligibility traces

Understanding SARSA (λ)

Exercises

Summary

Section 2: Exploiting the Knowledge

Section 2: Exploiting the Knowledge

Going Deep with DQN

Going Deep with DQN

DL for RL

Using PyTorch for DL

Building neural networks with Torch

Understanding DQN in PyTorch

Exercising DQN

Exercises

Summary

Going Deeper with DDQN

Going Deeper with DDQN

Understanding visual state

Introducing CNNs

Working with a DQN on Atari

Introducing DDQN

Extending replay with prioritized experience replay

Exercises

Summary

Policy Gradient Methods

Policy Gradient Methods

Understanding policy gradient methods

Introducing REINFORCE

Using advantage actor-critic

Building a deep deterministic policy gradient

Exploring trust region policy optimization

Exercises

Summary

Optimizing for Continuous Control

Optimizing for Continuous Control

Understanding continuous control with Mujoco

Introducing proximal policy optimization

Using PPO with recurrent networks

Deciding on synchronous and asynchronous actors

Building actor-critic with experience replay

Exercises

Summary

All about Rainbow DQN

All about Rainbow DQN

Rainbow – combining improvements in deep reinforcement learning

Using TensorBoard

Introducing distributional RL

Understanding noisy networks

Unveiling Rainbow DQN

Exercises

Summary

Exploiting ML-Agents

Exploiting ML-Agents

Installing ML-Agents

Building a Unity environment

Training a Unity environment with Rainbow

Creating a new environment

Advancing RL with ML-Agents

Exercises

Summary

DRL Frameworks

DRL Frameworks

Choosing a framework

Introducing Google Dopamine

Playing with Keras-RL

Exploring RL Lib

Using TF-Agents

Exercises

Summary

Section 3: Reward Yourself

Section 3: Reward Yourself

3D Worlds

3D Worlds

Reasoning on 3D worlds

Training a visual agent

Generalizing 3D vision

Challenging the Unity Obstacle Tower Challenge

Exploring Habitat – embodied agents by FAIR

Exercises

Summary

From DRL to AGI

From DRL to AGI

Learning meta learning

Introducing meta reinforcement learning

Using hindsight experience replay

Imagination and reasoning in RL

Understanding imagination-augmented agents

Exercises

Summary

Other Books You May Enjoy

Other Books You May Enjoy

Leave a review - let other readers know what you think

Customer Reviews

5 (3)

5 star

100%

4 star

0

3 star

0

2 star

0

1 star

0

Working with TD (λ) and eligibility traces

Up until now, we have looked at the forward view or what the agent perceives to be as the next best reward or state. In MC, we looked at the entire episode and then used those values to reverse calculate returns. For TDL methods such as Q-learning and SARSA, we looked a single step ahead or what we referred to as TD (0). However, we want our agents to be able to take into account several steps, n, in advance. If we can do this, then surely our agent will be able to make better decisions.

As we have seen previously, we can average returns across steps using a discount factor, gamma. However, at this point, we need to more careful about how we average or collect returns. Instead, we can define the averaging of all returns over an infinite number of steps forward as follows:

In the preceding equation, we have the following:

This...

Search

Your notes and bookmarks