AnnaMats/ppo-Pyramids-Training

1# Table of Contents2 3* [mlagents.trainers.optimizer.torch\_optimizer](#mlagents.trainers.optimizer.torch_optimizer)4  * [TorchOptimizer](#mlagents.trainers.optimizer.torch_optimizer.TorchOptimizer)5    * [create\_reward\_signals](#mlagents.trainers.optimizer.torch_optimizer.TorchOptimizer.create_reward_signals)6    * [get\_trajectory\_value\_estimates](#mlagents.trainers.optimizer.torch_optimizer.TorchOptimizer.get_trajectory_value_estimates)7* [mlagents.trainers.optimizer.optimizer](#mlagents.trainers.optimizer.optimizer)8  * [Optimizer](#mlagents.trainers.optimizer.optimizer.Optimizer)9    * [update](#mlagents.trainers.optimizer.optimizer.Optimizer.update)10 11<a name="mlagents.trainers.optimizer.torch_optimizer"></a>12# mlagents.trainers.optimizer.torch\_optimizer13 14<a name="mlagents.trainers.optimizer.torch_optimizer.TorchOptimizer"></a>15## TorchOptimizer Objects16 17```python18class TorchOptimizer(Optimizer)19```20 21<a name="mlagents.trainers.optimizer.torch_optimizer.TorchOptimizer.create_reward_signals"></a>22#### create\_reward\_signals23 24```python25 | create_reward_signals(reward_signal_configs: Dict[RewardSignalType, RewardSignalSettings]) -> None26```27 28Create reward signals29 30**Arguments**:31 32- `reward_signal_configs`: Reward signal config.33 34<a name="mlagents.trainers.optimizer.torch_optimizer.TorchOptimizer.get_trajectory_value_estimates"></a>35#### get\_trajectory\_value\_estimates36 37```python38 | get_trajectory_value_estimates(batch: AgentBuffer, next_obs: List[np.ndarray], done: bool, agent_id: str = "") -> Tuple[Dict[str, np.ndarray], Dict[str, float], Optional[AgentBufferField]]39```40 41Get value estimates and memories for a trajectory, in batch form.42 43**Arguments**:44 45- `batch`: An AgentBuffer that consists of a trajectory.46- `next_obs`: the next observation (after the trajectory). Used for boostrapping47    if this is not a termiinal trajectory.48- `done`: Set true if this is a terminal trajectory.49- `agent_id`: Agent ID of the agent that this trajectory belongs to.50 51**Returns**:52 53A Tuple of the Value Estimates as a Dict of [name, np.ndarray(trajectory_len)],54    the final value estimate as a Dict of [name, float], and optionally (if using memories)55    an AgentBufferField of initial critic memories to be used during update.56 57<a name="mlagents.trainers.optimizer.optimizer"></a>58# mlagents.trainers.optimizer.optimizer59 60<a name="mlagents.trainers.optimizer.optimizer.Optimizer"></a>61## Optimizer Objects62 63```python64class Optimizer(abc.ABC)65```66 67Creates loss functions and auxillary networks (e.g. Q or Value) needed for training.68Provides methods to update the Policy.69 70<a name="mlagents.trainers.optimizer.optimizer.Optimizer.update"></a>71#### update72 73```python74 | @abc.abstractmethod75 | update(batch: AgentBuffer, num_sequences: int) -> Dict[str, float]76```77 78Update the Policy based on the batch that was passed in.79 80**Arguments**:81 82- `batch`: AgentBuffer that contains the minibatch of data used for this update.83- `num_sequences`: Number of recurrent sequences found in the minibatch.84 85**Returns**:86 87A Dict containing statistics (name, value) from the update (e.g. loss)88