To establish the baseline results, we'll use the A2C method, in a very similar way to the code in the previous chapter. The complete source is in files Chapter15/01_train_a2c.py and Chapter15/lib/model.py. There are a few differences between this baseline and version we've used in the previous chapter. First of all, there are 16 parallel environments used to gather the experience during the training. The second difference is the model structure and the way that we perform exploration. To illustrate them, let's look at the model and the agent classes.

Both the actor and critic are placed in the separate networks without sharing weights. They follow the approach used in the previous chapter, when our critic estimates the mean and the variance for the actions, but now, variance is not a separate head of the base network, but just a single parameter of the model. This parameter will be adjusted during the training by SGD, but it doesn't depend...