openenv/atari_env

1Metadata-Version: 2.42Name: openenv-core3Version: 0.2.34Summary: A unified framework for reinforcement learning environments5Requires-Python: >=3.106Description-Content-Type: text/markdown7License-File: LICENSE8Requires-Dist: fastapi>=0.104.09Requires-Dist: pydantic>=2.0.010Requires-Dist: uvicorn>=0.24.011Requires-Dist: requests>=2.25.012Requires-Dist: typer>=0.9.013Requires-Dist: rich>=13.0.014Requires-Dist: pyyaml>=6.015Requires-Dist: huggingface_hub>=0.20.016Requires-Dist: openai>=2.7.217Requires-Dist: tomli>=2.3.018Requires-Dist: tomli-w>=1.2.019Requires-Dist: websockets>=15.0.120Requires-Dist: fastmcp>=3.0.021Requires-Dist: gradio>=4.0.022Requires-Dist: httpx>=0.28.123Provides-Extra: core24Requires-Dist: fastapi>=0.104.0; extra == "core"25Requires-Dist: pydantic>=2.0.0; extra == "core"26Requires-Dist: uvicorn>=0.24.0; extra == "core"27Requires-Dist: requests>=2.25.0; extra == "core"28Requires-Dist: websockets>=15.0.1; extra == "core"29Provides-Extra: cli30Requires-Dist: typer>=0.9.0; extra == "cli"31Requires-Dist: rich>=13.0.0; extra == "cli"32Requires-Dist: pyyaml>=6.0; extra == "cli"33Requires-Dist: huggingface_hub>=0.20.0; extra == "cli"34Requires-Dist: openai>=2.7.2; extra == "cli"35Requires-Dist: tomli>=2.3.0; extra == "cli"36Requires-Dist: tomli-w>=1.2.0; extra == "cli"37Provides-Extra: docs38Requires-Dist: sphinx==7.2.6; extra == "docs"39Requires-Dist: pytorch-sphinx-theme2; extra == "docs"40Requires-Dist: sphinxcontrib.katex==0.9.10; extra == "docs"41Requires-Dist: docutils<0.21,>=0.18.1; extra == "docs"42Requires-Dist: sphinx-design==0.6.1; extra == "docs"43Requires-Dist: sphinxcontrib-mermaid==1.0.0; extra == "docs"44Requires-Dist: myst-parser; extra == "docs"45Requires-Dist: sphinxext-opengraph; extra == "docs"46Requires-Dist: sphinx-sitemap==2.7.1; extra == "docs"47Requires-Dist: sphinx-gallery>=0.14.0; extra == "docs"48Requires-Dist: matplotlib; extra == "docs"49Requires-Dist: nest-asyncio; extra == "docs"50Requires-Dist: smolagents; extra == "docs"51Provides-Extra: all52Requires-Dist: openenv-core[core]; extra == "all"53Requires-Dist: openenv-core[cli]; extra == "all"54Provides-Extra: daytona55Requires-Dist: daytona>=0.136.0; extra == "daytona"56Requires-Dist: pyyaml>=6.0; extra == "daytona"57Provides-Extra: inspect58Requires-Dist: inspect-ai>=0.3.0; extra == "inspect"59Dynamic: license-file60 61# <img width="35" height="35" alt="image" src="https://github.com/user-attachments/assets/2700a971-e5d6-4036-b03f-2f89c9791609" /> OpenEnv: Agentic Execution Environments62 63An e2e framework for creating, deploying and using isolated execution environments for agentic RL training, built using Gymnasium style simple APIs.64 65[![PyPI](https://img.shields.io/pypi/v/openenv-core?color=blue)](https://pypi.org/project/openenv-core/)66[![Discord](https://img.shields.io/badge/Discord-OpenEnv-7289da?style=flat&logo=discord&logoColor=white)](https://discord.gg/YsTYBh6PD9)67[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/OpenEnv_Tutorial.ipynb)68[![Docs](https://img.shields.io/badge/Docs-Explore-blue?logo=readthedocs&logoColor=white)](https://meta-pytorch.org/OpenEnv/)69 70---71 72**🚀 Featured Example:** Train LLMs to play BlackJack using [torchforge](https://github.com/meta-pytorch/torchforge) (PyTorch's agentic RL framework): [`examples/grpo_blackjack/`](examples/grpo_blackjack/)73 74**🔥 Zero to Hero Tutorial:** End to end tutorial from our [GPU Mode](tutorial/README.md) lecture and other hackathons.75 76## Quick Start77 78Install the OpenEnv core package:79 80```bash81pip install openenv-core82```83 84Install an environment client (e.g., Echo):85 86```bash87pip install git+https://huggingface.co/spaces/openenv/echo_env88```89 90Then use the environment:91 92```python93import asyncio94from echo_env import EchoAction, EchoEnv95 96async def main():97    # Connect to a running Space (async context manager)98    async with EchoEnv(base_url="https://openenv-echo-env.hf.space") as client:99        # Reset the environment100        result = await client.reset()101        print(result.observation.echoed_message)  # "Echo environment ready!"102 103        # Send messages104        result = await client.step(EchoAction(message="Hello, World!"))105        print(result.observation.echoed_message)  # "Hello, World!"106        print(result.reward)  # 1.3 (based on message length)107 108asyncio.run(main())109```110 111**Synchronous usage** is also supported via the `.sync()` wrapper:112 113```python114from echo_env import EchoAction, EchoEnv115 116# Use .sync() for synchronous context manager117with EchoEnv(base_url="https://openenv-echo-env.hf.space").sync() as client:118    result = client.reset()119    result = client.step(EchoAction(message="Hello, World!"))120    print(result.observation.echoed_message)121```122 123For a detailed quick start, check out the [docs page](https://meta-pytorch.org/OpenEnv/quickstart/).124 125## OpenEnv on partner platforms:126 127- [Lightning AI Studio](https://lightning.ai/environments?section=featured)128- [TRL example](https://huggingface.co/docs/trl/openenv)129- [Unsloth Google Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/OpenEnv_gpt_oss_(20B)_Reinforcement_Learning_2048_Game.ipynb)130- [ART example](https://art.openpipe.ai/integrations/openenv-integration)131- [Oumi example](https://github.com/oumi-ai/oumi/blob/main/notebooks/Oumi%20-%20OpenEnv%20GRPO%20with%20trl.ipynb)132 133## Overview134 135OpenEnv provides a standard for interacting with agentic execution environments via simple Gymnasium style APIs - `step()`, `reset()`, `state()`. Users of agentic execution environments can interact with the environment during RL training loops using these simple APIs.136 137In addition to making it easier for researchers and RL framework writers, we also provide tools for environment creators making it easier for them to create richer environments and make them available over familiar protocols like HTTP and packaged using canonical technologies like docker. Environment creators can use the OpenEnv framework to create environments that are isolated, secure, and easy to deploy and use.138 139The OpenEnv CLI (`openenv`) provides commands to initialize new environments and deploy them to Hugging Face Spaces.140 141> ⚠️ **Early Development Warning** OpenEnv is currently in an experimental142> stage. You should expect bugs, incomplete features, and APIs that may change143> in future versions. The project welcomes bugfixes, but to make sure things are144> well coordinated you should discuss any significant change before starting the145> work. It's recommended that you signal your intention to contribute in the146> issue tracker, either by filing a new issue or by claiming an existing one.147 148### RFCs149 150Below is a list of active and historical RFCs for OpenEnv. RFCs are proposals for major changes or features. Please review and contribute!151 152- [RFC 001: Baseline API and Interface Specifications](https://github.com/meta-pytorch/OpenEnv/pull/26)153- [RFC 002: Discoverability of environment tools by agents](https://github.com/meta-pytorch/OpenEnv/pull/32)154- [RFC 003: Add MCP (Model Context Protocol) support](https://github.com/meta-pytorch/OpenEnv/pull/224)155- [RFC 004: Add delayed rewards support for trajectory-based scoring](https://github.com/meta-pytorch/OpenEnv/pull/337)156- [RFC 005: Agentic Harness Integration](https://github.com/meta-pytorch/OpenEnv/pull/387)157 158## Architecture159 160### Component Overview161 162```163┌─────────────────────────────────────────────────────────┐164│                    Client Application                   │165│  ┌────────────────┐              ┌──────────────────┐   │166│  │  EchoEnv       │              │  CodingEnv       │   │167│  │  (EnvClient)   │              │   (EnvClient)    │   │168│  └────────┬───────┘              └────────┬─────────┘   │169└───────────┼───────────────────────────────┼─────────────┘170            │ WebSocket                     │ WebSocket171            │ (reset, step, state)          │172┌───────────▼───────────────────────────────▼─────────────┐173│              Docker Containers (Isolated)               │174│  ┌──────────────────────┐    ┌──────────────────────┐   │175│  │ FastAPI Server       │    │ FastAPI Server       │   │176│  │   EchoEnvironment    │    │ PythonCodeActEnv     │   │177│  │ (Environment base)   │    │ (Environment base)   │   │178│  └──────────────────────┘    └──────────────────────┘   │179└─────────────────────────────────────────────────────────┘180```181 182### Core Components183 184#### 1. Web Interface185 186OpenEnv includes a built-in web interface for interactive environment exploration and debugging. The web interface provides:187 188- **Two-Pane Layout**: HumanAgent interaction on the left, state observation on the right189- **Real-time Updates**: WebSocket-based live updates without page refresh190- **Dynamic Forms**: Automatically generated action forms based on environment Action types191- **Action History**: Complete log of all actions taken and their results192 193The web interface is **conditionally enabled** based on environment variables:194 195- **Local Development**: Disabled by default for lightweight development196- **Manual Override**: Enable with `ENABLE_WEB_INTERFACE=true`197 198To use the web interface:199 200```python201from openenv.core.env_server import create_web_interface_app202from your_env.models import YourAction, YourObservation203from your_env.server.your_environment import YourEnvironment204 205env = YourEnvironment()206app = create_web_interface_app(env, YourAction, YourObservation)207```208 209When enabled, open `http://localhost:8000/web` in your browser to interact with the environment.210 211#### 2. Environment (Server-Side)212Base class for implementing environment logic:213- **`reset()`**: Initialize a new episode, returns initial `Observation`214- **`step(action)`**: Execute an `Action`, returns resulting `Observation`215- **`state()`**: Access episode metadata (`State` with episode_id, step_count, etc.)216 217#### 3. EnvClient (Client-Side)218Base class for environment communication:219- **Async by default**: Use `async with` and `await` for all operations220- **Sync wrapper**: Call `.sync()` to get a `SyncEnvClient` for synchronous usage221- Handles WebSocket connections to environment server222- Contains a utility to spin up a docker container locally for the corresponding environment223- Type-safe action/observation parsing224 225#### 4. Container Providers226Manage container deployment:227- `LocalDockerProvider`: Run containers on local Docker daemon228- `KubernetesProvider`: Deploy to K8s clusters (future)229 230#### 5. Models231Type-safe data structures:232- `Action`: Base class for environment actions233- `Observation`: Base class for environment observations234- `State`: Episode state tracking235- `StepResult`: Combines observation, reward, done flag236 237## Project Structure238 239### For Environment Creators240 241Use the CLI to quickly scaffold a new environment:242 243```bash244openenv init my_env245```246 247This creates the following structure:248 249```250my_env/251├── .dockerignore        # Docker build exclusions252├── __init__.py           # Export YourAction, YourObservation, YourEnv253├── models.py             # Define Action, Observation, State dataclasses254├── client.py             # Implement YourEnv(EnvClient)255├── README.md             # Document your environment256├── openenv.yaml          # Environment manifest257├── pyproject.toml        # Dependencies and package configuration258├── outputs/              # Runtime outputs (logs, evals) - gitignored259│   ├── logs/260│   └── evals/261└── server/262    ├── your_environment.py  # Implement YourEnvironment(Environment)263    ├── app.py               # Create FastAPI app264    ├── requirements.txt     # Dependencies for Docker (can be generated)265    └── Dockerfile           # Define container image266```267 268#### Dependency Management269 270OpenEnv uses `pyproject.toml` as the primary dependency specification:271 272- **Environment-level `pyproject.toml`**: Each environment defines its own dependencies273- **Root-level `pyproject.toml`**: Contains shared core dependencies (fastapi, pydantic, uvicorn)274- **Server `requirements.txt`**: Can be auto-generated from `pyproject.toml` for Docker builds275 276**Development Workflow:**277 278```bash279# Install environment in editable mode280cd my_env281pip install -e .282 283# Or using uv (faster)284uv pip install -e .285 286# Run server locally without Docker287uv run server --host 0.0.0.0 --port 8000288```289 290**Benefits:**291- ✅ **Client-side extensions**: Modify client classes locally without repo changes292- ✅ **Better dependency management**: Clear separation between environments293- ✅ **Flexible workflows**: Use pip, uv, or Docker for different scenarios294- ✅ **CI/CD ready**: Automated dependency generation and validation295 296See [`envs/README.md`](envs/README.md) for a complete guide on building environments.297 298### For Environment Users299 300To use an environment:3011. Install the client: `pip install git+https://huggingface.co/spaces/openenv/echo-env`3022. Import: `from echo_env import EchoAction, EchoEnv`3033. Use async (recommended) or sync API:304 305**Async (recommended):**306```python307async with EchoEnv(base_url="...") as client:308    result = await client.reset()309    result = await client.step(action)310```311 312**Sync (via `.sync()` wrapper):**313```python314with EchoEnv(base_url="...").sync() as client:315    result = client.reset()316    result = client.step(action)317```318 319See example scripts in `examples/` directory.320 321## CLI Commands322 323The OpenEnv CLI provides commands to manage environments:324 325- **`openenv init <env_name>`** - Initialize a new environment from template326- **`openenv push [--repo-id <repo>] [--private]`** - Deploy environment to Hugging Face Spaces327 328### Quick Start329 330```bash331# Create a new environment332openenv init my_game_env333 334# Deploy to Hugging Face (will prompt for login if needed)335cd my_game_env336openenv push337```338 339For detailed options: `openenv init --help` and `openenv push --help`.340 341## Design Principles342 3431. **Separation of Concerns**: Clear client-server boundaries3442. **Type Safety**: Strongly-typed actions, observations, and state3453. **Container Isolation**: Each environment runs in its own container3464. **Simple APIs**: Minimal, intuitive interfaces347 348## Development349 350### Installation351 352```bash353# Clone the repository354git clone https://github.com/meta-pytorch/OpenEnv.git355cd OpenEnv356 357# Install core package in editable mode358pip install -e .359# Or using uv (faster)360uv pip install -e .361```362 363### Running Tests364 365OpenEnv uses a modular dependency structure: the core package is minimal, and each environment has its own dependencies. This means some tests require environment-specific packages.366 367```bash368# Install pytest (required for running tests)369uv pip install pytest370 371# Run all tests (skips tests requiring uninstalled dependencies)372PYTHONPATH=src:envs uv run pytest tests/ -v --tb=short373 374# Run a specific test file375PYTHONPATH=src:envs uv run pytest tests/envs/test_echo_environment.py -v376```377 378**To run environment-specific tests**, install that environment's dependencies:379 380```bash381# Example: Install coding_env with dev dependencies (includes smolagents + pytest)382uv pip install -e "envs/coding_env[dev]"383 384# Then run coding_env tests385PYTHONPATH=src:envs uv run pytest tests/envs/test_python_codeact_rewards.py -v386```387 388Tests will be automatically skipped if their required dependencies aren't installed.389 390## Requirements391 392- Python 3.10+393- Docker Desktop or Docker Engine394- FastAPI >= 0.104.0395- Uvicorn >= 0.24.0396- Requests >= 2.25.0397- Environment-specific dependencies (e.g., smolagents for coding_env)398 399## Supported RL Tools400The goal of this project is to support a broad set of open and closed tools to help standardize the agentic RL community. If you have a project that supports OpenEnv environments, please put up a PR to add your tool name along with a link to your documentation.401 402### torchforge403See GRPO BlackJack training example: [`examples/grpo_blackjack/`](examples/grpo_blackjack/)404 405### TRL406See the [TRL example](https://huggingface.co/docs/trl/openenv) on how to integrate OpenEnv environments with GRPO training.407 408### Unsloth409See the 2048 game example based on gpt-oss: [Colab notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/OpenEnv_gpt_oss_(20B)_Reinforcement_Learning_2048_Game.ipynb)410 411### SkyRL412See the [SkyRL example](https://skyrl.readthedocs.io/en/latest/examples/openenv.html) on how to train on OpenEnv environments with SkyRL.413 414### ART415See the [ART example](https://art.openpipe.ai/integrations/openenv-integration) on how OpenEnv environments can be used to train models with ART.416 417### Oumi418See the [Oumi example](https://github.com/oumi-ai/oumi/blob/main/notebooks/Oumi%20-%20OpenEnv%20GRPO%20with%20trl.ipynb) on how OpenEnv environments can be used to train models with Oumi.419 420## Example Environments421 422| Environment | Description |423|---|---|424| [Echo Environment](envs/echo_env/README.md) | Echoes back messages with metadata. Ideal for testing HTTP server infrastructure, learning framework basics, and verifying container deployment. |425| [Coding Environment](envs/coding_env/README.md) | Sandboxed Python code execution via smolagents. Captures stdout/stderr/exit codes, supports persistent episode context, and provides detailed error handling. |426| [Chess Environment](envs/chess_env/README.md) | Chess RL environment with configurable opponents and full rules support. |427| [Atari Environment](envs/atari_env/README.md) | Classic Arcade Learning Environment tasks for RL benchmarking. |428| [FinRL Environment](envs/finrl_env/README.md) | Financial market simulations for algorithmic trading experiments. |429 430> Browse the full catalog of community environments at [meta-pytorch.org/OpenEnv/environments](https://meta-pytorch.org/OpenEnv/environments/).431 432## Community Support & Acknowledgments433This is an open and community-centric project. If you would like to add your name here, please put up a pull request and tag @jspisak for review. Ty!!434 435Supporters include: Meta-PyTorch, Hugging Face, [Scaler AI Labs](https://scalerailabs.com), [Patronus AI](https://patronus.ai), [Surge AI](https://surgehq.ai), [LastMile AI](https://www.lastmileai.dev), Unsloth AI, Reflection AI, vLLM, SkyRL (UC-Berkeley), LightningAI, Axolotl AI, Stanford Scaling Intelligence Lab, Mithril, [OpenMined](https://openmined.org/), [Fleet AI](https://fleetai.com), [Halluminate](https://halluminate.ai/), [Turing](https://www.turing.com/), [Scale AI](https://scale.com/) ..436 437And we'd also like to acknowledge the team at Farama Foundation as the OpenEnv API was heavily inspired by the work you all have done on Gymnasium. Cheers!438 439## License440 441BSD 3-Clause License (see [LICENSE](./LICENSE) file)442