openenv/coding_env
21
1Metadata-Version: 2.42Name: openenv3Version: 0.2.04Summary: A unified framework for reinforcement learning environments5Requires-Python: >=3.106Description-Content-Type: text/markdown7License-File: LICENSE8Requires-Dist: fastapi>=0.104.09Requires-Dist: pydantic>=2.0.010Requires-Dist: uvicorn>=0.24.011Requires-Dist: requests>=2.25.012Requires-Dist: typer>=0.9.013Requires-Dist: rich>=13.0.014Requires-Dist: pyyaml>=6.015Requires-Dist: huggingface_hub>=0.20.016Requires-Dist: openai>=2.7.217Requires-Dist: tomli>=2.3.018Requires-Dist: tomli-w>=1.2.019Requires-Dist: websockets>=15.0.120Provides-Extra: core21Requires-Dist: fastapi>=0.104.0; extra == "core"22Requires-Dist: pydantic>=2.0.0; extra == "core"23Requires-Dist: uvicorn>=0.24.0; extra == "core"24Requires-Dist: requests>=2.25.0; extra == "core"25Requires-Dist: websockets>=15.0.1; extra == "core"26Provides-Extra: cli27Requires-Dist: typer>=0.9.0; extra == "cli"28Requires-Dist: rich>=13.0.0; extra == "cli"29Requires-Dist: pyyaml>=6.0; extra == "cli"30Requires-Dist: huggingface_hub>=0.20.0; extra == "cli"31Requires-Dist: openai>=2.7.2; extra == "cli"32Requires-Dist: tomli>=2.3.0; extra == "cli"33Requires-Dist: tomli-w>=1.2.0; extra == "cli"34Provides-Extra: all35Requires-Dist: openenv[core]; extra == "all"36Requires-Dist: openenv[cli]; extra == "all"37Dynamic: license-file38 39# <img width="35" height="35" alt="image" src="https://github.com/user-attachments/assets/2700a971-e5d6-4036-b03f-2f89c9791609" /> OpenEnv: Agentic Execution Environments40 41An e2e framework for creating, deploying and using isolated execution environments for agentic RL training, built using Gymnasium style simple APIs.42 43[](https://pypi.org/project/openenv/)44[](https://discord.gg/YsTYBh6PD9)45[](https://colab.research.google.com/github/meta-pytorch/OpenEnv/blob/main/examples/OpenEnv_Tutorial.ipynb)46[](https://meta-pytorch.org/OpenEnv/)47 48---49 50**🚀 Featured Example:** Train LLMs to play BlackJack using [torchforge](https://github.com/meta-pytorch/torchforge) (PyTorch's agentic RL framework): [`examples/grpo_blackjack/`](examples/grpo_blackjack/)51 52## OpenEnv on partner platforms:53 54- [Lightning AI Studio](https://lightning.ai/environments?section=featured)55- [TRL example](https://huggingface.co/docs/trl/main/en/openenv)56- [Unsloth Google Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/OpenEnv_gpt_oss_(20B)_Reinforcement_Learning_2048_Game.ipynb)57- [ART example](https://art.openpipe.ai/integrations/openenv-integration)58- [Oumi example](https://github.com/oumi-ai/oumi/blob/main/notebooks/Oumi%20-%20OpenEnv%20GRPO%20with%20trl.ipynb)59 60## Overview61 62OpenEnv provides a standard for interacting with agentic execution environments via simple Gymnasium style APIs - `step()`, `reset()`, `state()`. Users of agentic execution environments can interact with the environment during RL training loops using these simple APIs.63 64In addition to making it easier for researchers and RL framework writers, we also provide tools for environment creators making it easier for them to create richer environments and make them available over familiar protocols like HTTP and packaged using canonical technologies like docker. Environment creators can use the OpenEnv framework to create environments that are isolated, secure, and easy to deploy and use.65 66The OpenEnv CLI (`openenv`) provides commands to initialize new environments and deploy them to Hugging Face Spaces.67 68> ⚠️ **Early Development Warning** OpenEnv is currently in an experimental69> stage. You should expect bugs, incomplete features, and APIs that may change70> in future versions. The project welcomes bugfixes, but to make sure things are71> well coordinated you should discuss any significant change before starting the72> work. It's recommended that you signal your intention to contribute in the73> issue tracker, either by filing a new issue or by claiming an existing one.74 75### RFCs76 77Below is a list of active and historical RFCs for OpenEnv. RFCs are proposals for major changes or features. Please review and contribute!78 79- [RFC 001: Baseline API and Interface Specifications](https://github.com/meta-pytorch/OpenEnv/pull/26)80 81## Architecture82 83### Component Overview84 85```86┌─────────────────────────────────────────────────────────┐87│ Client Application │88│ ┌────────────────┐ ┌──────────────────┐ │89│ │ EchoEnv │ │ CodingEnv │ │90│ │ (HTTPEnvClient)│ │ (HTTPEnvClient) │ │91│ └────────┬───────┘ └────────┬─────────┘ │92└───────────┼───────────────────────────────┼─────────────┘93 │ HTTP │ HTTP94 │ (reset, step, state) │95┌───────────▼───────────────────────────────▼─────────────┐96│ Docker Containers (Isolated) │97│ ┌──────────────────────┐ ┌──────────────────────┐ │98│ │ FastAPI Server │ │ FastAPI Server │ │99│ │ EchoEnvironment │ │ PythonCodeActEnv │ │100│ │ (Environment base) │ │ (Environment base) │ │101│ └──────────────────────┘ └──────────────────────┘ │102└─────────────────────────────────────────────────────────┘103```104 105### Core Components106 107#### 1. Web Interface108 109OpenEnv includes a built-in web interface for interactive environment exploration and debugging. The web interface provides:110 111- **Two-Pane Layout**: HumanAgent interaction on the left, state observation on the right112- **Real-time Updates**: WebSocket-based live updates without page refresh113- **Dynamic Forms**: Automatically generated action forms based on environment Action types114- **Action History**: Complete log of all actions taken and their results115 116The web interface is **conditionally enabled** based on environment variables:117 118- **Local Development**: Disabled by default for lightweight development119- **Manual Override**: Enable with `ENABLE_WEB_INTERFACE=true`120 121To use the web interface:122 123```python124from openenv.core.env_server import create_web_interface_app125from your_env.models import YourAction, YourObservation126from your_env.server.your_environment import YourEnvironment127 128env = YourEnvironment()129app = create_web_interface_app(env, YourAction, YourObservation)130```131 132When enabled, open `http://localhost:8000/web` in your browser to interact with the environment.133 134#### 2. Environment (Server-Side)135Base class for implementing environment logic:136- **`reset()`**: Initialize a new episode, returns initial `Observation`137- **`step(action)`**: Execute an `Action`, returns resulting `Observation`138- **`state()`**: Access episode metadata (`State` with episode_id, step_count, etc.)139 140#### 3. HTTPEnvClient (Client-Side)141Base class for HTTP communication:142- Handles HTTP requests to environment server143- Contains a utility to spin up a docker container locally for the corresponding environment144- Type-safe action/observation parsing145 146#### 4. Container Providers147Manage container deployment:148- `LocalDockerProvider`: Run containers on local Docker daemon149- `KubernetesProvider`: Deploy to K8s clusters (future)150 151#### 5. Models152Type-safe data structures:153- `Action`: Base class for environment actions154- `Observation`: Base class for environment observations155- `State`: Episode state tracking156- `StepResult`: Combines observation, reward, done flag157 158## Project Structure159 160### For Environment Creators161 162Use the CLI to quickly scaffold a new environment:163 164```bash165openenv init my_env166```167 168This creates the following structure:169 170```171my_env/172├── .dockerignore # Docker build exclusions173├── __init__.py # Export YourAction, YourObservation, YourEnv174├── models.py # Define Action, Observation, State dataclasses175├── client.py # Implement YourEnv(HTTPEnvClient)176├── README.md # Document your environment177├── openenv.yaml # Environment manifest178├── pyproject.toml # Dependencies and package configuration179├── outputs/ # Runtime outputs (logs, evals) - gitignored180│ ├── logs/181│ └── evals/182└── server/183 ├── your_environment.py # Implement YourEnvironment(Environment)184 ├── app.py # Create FastAPI app185 ├── requirements.txt # Dependencies for Docker (can be generated)186 └── Dockerfile # Define container image187```188 189#### Dependency Management190 191OpenEnv uses `pyproject.toml` as the primary dependency specification:192 193- **Environment-level `pyproject.toml`**: Each environment defines its own dependencies194- **Root-level `pyproject.toml`**: Contains shared core dependencies (fastapi, pydantic, uvicorn)195- **Server `requirements.txt`**: Can be auto-generated from `pyproject.toml` for Docker builds196 197**Development Workflow:**198 199```bash200# Install environment in editable mode201cd my_env202pip install -e .203 204# Or using uv (faster)205uv pip install -e .206 207# Run server locally without Docker208uv run server --host 0.0.0.0 --port 8000209```210 211**Benefits:**212- ✅ **Client-side extensions**: Modify client classes locally without repo changes213- ✅ **Better dependency management**: Clear separation between environments214- ✅ **Flexible workflows**: Use pip, uv, or Docker for different scenarios215- ✅ **CI/CD ready**: Automated dependency generation and validation216 217See [`envs/README.md`](envs/README.md) for a complete guide on building environments.218 219### For Environment Users220 221To use an environment:2221. Import from `envs.your_env`: `from envs.echo_env import EchoAction, EchoEnv`2232. Create client: `client = EchoEnv.from_docker_image("echo-env:latest")`2243. Interact: `client.reset()`, `client.step(action)`, `client.state()`2254. Cleanup: `client.close()`226 227See example scripts in `examples/` directory.228 229## CLI Commands230 231The OpenEnv CLI provides commands to manage environments:232 233- **`openenv init <env_name>`** - Initialize a new environment from template234- **`openenv push [--repo-id <repo>] [--private]`** - Deploy environment to Hugging Face Spaces235 236### Quick Start237 238```bash239# Create a new environment240openenv init my_game_env241 242# Deploy to Hugging Face (will prompt for login if needed)243cd my_game_env244openenv push245```246 247For detailed options: `openenv init --help` and `openenv push --help`.248 249## Design Principles250 2511. **Separation of Concerns**: Clear client-server boundaries2522. **Type Safety**: Strongly-typed actions, observations, and state2533. **Container Isolation**: Each environment runs in its own container2544. **Simple APIs**: Minimal, intuitive interfaces255 256## Quick Start257 258### Using the Echo Environment(Example)259 260```python261from envs.echo_env import EchoAction, EchoEnv262 263# Automatically start container and connect264client = EchoEnv.from_docker_image("echo-env:latest")265 266# Reset the environment267result = client.reset()268print(result.observation.echoed_message) # "Echo environment ready!"269 270# Send messages271result = client.step(EchoAction(message="Hello, World!"))272print(result.observation.echoed_message) # "Hello, World!"273print(result.reward) # 1.3 (based on message length)274 275# Cleanup276client.close() # Stops and removes container277```278 279## Requirements280 281- Python 3.11+282- Docker Desktop or Docker Engine283- FastAPI >= 0.104.0284- Uvicorn >= 0.24.0285- Requests >= 2.25.0286- smolagents (for coding environment)287 288## Supported RL Tools289The goal of this project is to support a broad set of open and closed tools to help standardize the agentic RL community. If you have a project that supports OpenEnv environments, please put up a PR to add your tool name along with a link to your documentation.290 291### torchforge292See GRPO BlackJack training example: [`examples/grpo_blackjack/`](examples/grpo_blackjack/)293 294### TRL295See the [TRL example](https://huggingface.co/docs/trl/main/en/openenv) on how to integrate OpenEnv environments with GRPO training.296 297### Unsloth298See the 2048 game example based on gpt-oss: [Colab notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/OpenEnv_gpt_oss_(20B)_Reinforcement_Learning_2048_Game.ipynb)299 300### SkyRL301See the [SkyRL example](https://skyrl.readthedocs.io/en/latest/examples/openenv.html) on how to train on OpenEnv environments with SkyRL.302 303### ART304See the [ART example](https://art.openpipe.ai/integrations/openenv-integration) on how OpenEnv environments can be used to train models with ART.305 306### Oumi307See the [Oumi example](https://github.com/oumi-ai/oumi/blob/main/notebooks/Oumi%20-%20OpenEnv%20GRPO%20with%20trl.ipynb) on how OpenEnv environments can be used to train models with Oumi.308 309## Example Environments310 311### Echo Environment312A simple environment that echoes back messages with metadata. Perfect for:313- Testing the HTTP server infrastructure314- Learning the framework basics315- Verifying container deployment316 317See: [`envs/echo_env/README.md`](envs/echo_env/README.md)318 319### Coding Environment320Executes arbitrary Python code in a sandboxed environment. Features:321- Safe code execution using smolagents322- Capture stdout, stderr, and exit codes323- Persistent execution context within episodes324- Error handling with detailed messages325 326See: [`envs/coding_env/README.md`](envs/coding_env/README.md)327 328## Community Support & Acknowledgments329This is an open and community-centric project. If you would like to add your name here, please put up a pull request and tag @jspisak for review. Ty!!330 331Supporters include: Meta-PyTorch, Hugging Face, [Patronus AI](https://patronus.ai), [Surge AI](https://surgehq.ai), [LastMile AI](https://www.lastmileai.dev), Unsloth AI, Reflection AI, vLLM, SkyRL (UC-Berkeley), LightningAI, Axolotl AI, Stanford Scaling Intelligence Lab, Mithril, [OpenMined](https://openmined.org/), [Fleet AI](https://fleetai.com), [Halluminate](https://halluminate.ai/), [Turing](https://www.turing.com/) ..332 333And we'd also like to acknowledge the team at Farama Foundation as the OpenEnv API was heavily inspired by the work you all have done on Gymnasium. Cheers!334 335## License336 337BSD 3-Clause License (see [LICENSE](./LICENSE) file)338