AnnaMats/ppo-Pyramids-Training

1# Unity Ml-Agents Custom trainers Plugin2 3As an attempt to bring a wider variety of reinforcement learning algorithms to our users, we have added custom trainers4capabilities. we introduce an extensible plugin system to define new trainers based on the High level trainer API5in `Ml-agents` Package. This will allow rerouting `mlagents-learn` CLI to custom trainers and extending the config files6with hyper-parameters specific to your new trainers. We will expose a high-level extensible trainer (both on-policy,7and off-policy trainers) optimizer and hyperparameter classes with documentation for the use of this plugin. For more8infromation on how python plugin system works see [Plugin interfaces](Training-Plugins.md).9## Overview10Model-free RL algorithms generally fall into two broad categories: on-policy and off-policy. On-policy algorithms perform updates based on data gathered from the current policy. Off-policy algorithms learn a Q function from a buffer of previous data, then use this Q function to make decisions. Off-policy algorithms have three key benefits in the context of ML-Agents: They tend to use fewer samples than on-policy as they can pull and re-use data from the buffer many times. They allow player demonstrations to be inserted in-line with RL data into the buffer, enabling new ways of doing imitation learning by streaming player data.11 12To add new custom trainers to ML-agents, you would need to create a new python package.13To give you an idea of how to structure your package, we have created a [mlagents_trainer_plugin](../ml-agents-trainer-plugin) package ourselves as an14example, with implementation of `A2c` and `DQN` algorithms. You would need a `setup.py` file to list extra requirements and15register the new RL algorithm in ml-agents ecosystem and be able to call `mlagents-learn` CLI with your customized16configuration.17 18 19```shell20├── mlagents_trainer_plugin21│    ├── __init__.py22│    ├── a2c23│    │    ├── __init__.py24│    │    ├── a2c_3DBall.yaml25│    │    ├── a2c_optimizer.py26│    │    └── a2c_trainer.py27│    └── dqn28│        ├── __init__.py29│        ├── dqn_basic.yaml30│        ├── dqn_optimizer.py31│        └── dqn_trainer.py32└── setup.py33```34## Installation and Execution35If you haven't already, follow the [installation instructions](Installation.md). Once you have the `ml-agents-env` and `ml-agents` packages you can install the plugin package. From the repository's root directory install `ml-agents-trainer-plugin` (or replace with the name of your plugin folder).36 37```sh38pip3 install -e <./ml-agents-trainer-plugin>39```40 41Following the previous installations your package is added as an entrypoint and you can use a config file with new42trainers:43```sh44mlagents-learn ml-agents-trainer-plugin/mlagents_trainer_plugin/a2c/a2c_3DBall.yaml --run-id <run-id-name>45--env <env-executable>46```47 48## Tutorial49Here’s a step-by-step [tutorial](Tutorial-Custom-Trainer-Plugin.md) on how to write a setup file and extend ml-agents trainers, optimizers, and50hyperparameter settings.To extend ML-agents classes see references on51[trainers](Python-On-Off-Policy-Trainer-Documentation.md) and [Optimizer](Python-Optimizer-Documentation.md).52