Proximal Policy Gradient (PPO)

Overview

PPO is one of the most popular DRL algorithms. It runs reasonably fast by leveraging vector (parallel) environments and naturally works well with different action spaces, therefore supporting a variety of games. It also has good sample efficiency compared to algorithms such as DQN.

Original paper:

• Proximal Policy Optimization Algorithms

Reference resources:

• Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study

All our PPO implementations below are augmented with the same code-level optimizations presented in openai/baselines's PPO. See The 32 Implementation Details of Proximal Policy Optimization (PPO) Algorithm for more details.

Implemented Variants

Variants Implemented	Description
ppo.py, docs	For classic control tasks like CartPole-v1.
ppo_atari.py, docs	For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
ppo_continuous_action.py, docs	For continuous action space. Also implemented Mujoco-specific code-level optimizations

Below are our single-file implementations of PPO:

ppo.py

The ppo.py has the following features:

• Works with the Box observation space of low-level features
• Works with the Discerete action space
• Works with envs like CartPole-v1

Usage

poetry install
python cleanrl/ppo.py --help
python cleanrl/ppo.py --env-id CartPole-v1

Implementation details

ppo.py includes the 11 core implementation details:

1. Vectorized architecture ( common/cmd_util.py#L22)
2. Orthogonal Initialization of Weights and Constant Initialization of biases ( a2c/utils.py#L58))
3. The Adam Optimizer's Epsilon Parameter ( ppo2/model.py#L100)
4. Adam Learning Rate Annealing ( ppo2/ppo2.py#L133-L135)
5. Generalized Advantage Estimation ( ppo2/runner.py#L56-L65)
6. Mini-batch Updates ( ppo2/ppo2.py#L157-L166)
7. Normalization of Advantages ( ppo2/model.py#L139)
8. Clipped surrogate objective ( ppo2/model.py#L81-L86)
9. Value Function Loss Clipping ( ppo2/model.py#L68-L75)
10. Overall Loss and Entropy Bonus ( ppo2/model.py#L91)
11. Global Gradient Clipping ( ppo2/model.py#L102-L108)

Experiment results

PR vwxyzjn/cleanrl#120 tracks our effort to conduct experiments, and the reprodudction instructions can be found at vwxyzjn/cleanrl/benchmark/ppo.

Below are the average episodic returns for ppo.py. To ensure the quality of the implementation, we compared the results against openai/baselies' PPO.

Environment	ppo.py	openai/baselies' PPO
CartPole-v1	488.75 ± 18.40	497.54 ± 4.02
Acrobot-v1	-82.48 ± 5.93	-81.82 ± 5.58
MountainCar-v0	-200.00 ± 0.00	-200.00 ± 0.00

Learning curves:

Tracked experiments and game play videos:

Video tutorial

If you'd like to learn ppo.py in-depth, consider checking out the following video tutorial:

ppo_atari.py

The ppo_atari.py has the following features:

• For playing Atari games. It uses convolutional layers and common atari-based pre-processing techniques.
• Works with the Atari's pixel Box observation space of shape (210, 160, 3)
• Works with the Discerete action space
• Includes the 9 Atari-specific implementation details as shown in the following video tutorial

ppo_continuous_action.py

The ppo_continuous_action.py has the following features:

• For continuous action space. Also implemented Mujoco-specific code-level optimizations
• Works with the Box observation space of low-level features
• Works with the Box (continuous) action space
• Includes the 8 implementation details for as shown in the following video tutorial (need fixing)