We will now understand the use of dueling network architectures. In DQN and DDQN, and other DQN variants in the literature, the focus was primarily on algorithms, that is, how to efficiently and stably update the value function neural networks. While this is crucial for developing robust RL algorithms, a parallel but complementary direction to advance the field is to also innovate and develop novel neural network architectures that are well suited for model-free RL. This is precisely the concept behind dueling network architectures, another contribution from DeepMind.

The steps involved in dueling architectures are as follows:

1. Dueling network architecture figure; compare with standard DQN
2. Computing Q(s,a)
3. Subtracting the average of the advantage from the advantage function

As we saw in the previous chapter, the output of the Q-network...