jester1177/cloud-native-debug-env
0
1---2title: Cloud-Native DevOps Debug Environment3emoji: 🔧4colorFrom: blue5colorTo: green6sdk: docker7app_port: 80008pinned: false9---10 11# Cloud-Native DevOps Debug Environment12 13An OpenEnv-compatible environment where AI agents learn to debug broken GitHub Actions workflows, Dockerfiles, and Kubernetes manifests. Built for the OpenEnv Hackathon by Scaler School of Technology (partners: Meta, HuggingFace, PyTorch).14 15## Why Cloud-Native Debugging?16 17Every developer who ships code hits deployment pipeline failures. A misconfigured Dockerfile, a broken GitHub Actions workflow, a missing secret, a Kubernetes selector mismatch — these are the bugs that waste hours of developer time every week. They're hard to debug because:18 19- Error messages are cryptic ("unable to prepare context: unable to evaluate symlinks")20- The feedback loop is slow (push, wait for CI, read logs, fix, repeat)21- Multiple config files interact in non-obvious ways (Dockerfile + workflow + secrets + K8s manifests)22- Kubernetes errors require cross-resource reasoning (Deployment labels must match Service selectors)23 24This environment teaches AI agents to do what senior DevOps engineers do: read the error, trace it to the root cause across multiple files, and fix it.25 26---27 28## How It Works: The Complete Flow29 30```31┌──────────────────────────────────────────────────────────────┐32│ 1. RESET │33│ Agent receives: │34│ - Broken config files (Dockerfile / workflow / K8s YAML) │35│ - Error message from the failed build/deploy │36│ - Available secrets list │37│ - Number of issues to find │38├──────────────────────────────────────────────────────────────┤39│ 2. OBSERVE → THINK → ACT (repeat up to 10 steps) │40│ Agent reads the error, analyzes the files, then: │41│ - edit_file: replace broken content with fixed content │42│ - replace_line: fix a specific line number │43│ - add_line / add_block: insert missing content │44│ - delete_line / delete_block: remove bad content │45│ - request_hint: get a clue (-5% score penalty) │46│ - submit: "I'm done fixing" │47│ │48│ After each action, agent gets: │49│ - Updated file contents │50│ - Reward signal (+0.3 per fix, -0.02 for failed edits) │51│ - How many issues are now fixed │52├──────────────────────────────────────────────────────────────┤53│ 3. GRADE │54│ Deterministic scoring based on: │55│ - What fraction of issues were fixed │56│ - Whether ALL issues were fixed (bonus) │57│ - How many steps it took (efficiency) │58│ - How many hints were used (penalty) │59└──────────────────────────────────────────────────────────────┘60```61 62---63 64## The 10 Tasks (50 Scenarios)65 66### Task 1: Dockerfile Syntax Errors — Easy67 68Simple typos and instruction errors that break `docker build`.69 70| # | Scenario | What's Broken | Real-World Context |71|---|----------|---------------|-------------------|72| 1 | `typo_filename` | `COPY requirments.txt .` — misspelled filename | Most common Docker build error on Stack Overflow |73| 2 | `invalid_base_image` | `FROM python:3.9-slimm` — extra 'm' in tag | Happens when copy-pasting image tags |74| 3 | `invalid_run_syntax` | `RUN pip install ... \n && python setup.py` — broken line continuation | Formatting multi-line RUN commands is tricky |75| 4 | `invalid_expose` | `EXPOSE "eighty"` — string instead of port number | EXPOSE only accepts numeric ports |76| 5 | `missing_from_instruction` | No `FROM` instruction at all | Dockerfile must start with FROM |77 78### Task 2: Dockerfile Runtime Errors — Medium79 80The Dockerfile builds successfully, but the container crashes at runtime.81 82| # | Scenario | What's Broken | Real-World Context |83|---|----------|---------------|-------------------|84| 1 | `missing_workdir` | No WORKDIR — files scatter to `/` | Container runs but `npm start` can't find `package.json` |85| 2 | `cmd_entrypoint_conflict` | Both ENTRYPOINT and CMD defined as full commands | Process starts incorrectly |86| 3 | `entrypoint_not_executable` | Shell script lacks execute permission | `chmod +x` missing — "permission denied" |87| 4 | `missing_required_env` | App needs `DATABASE_URL` but it's not set | Container crashes: "DATABASE_URL is not defined" |88| 5 | `non_root_privileged_port` | Non-root user tries to bind port 80 | Security best practice conflicts with port < 1024 |89 90### Task 3: Workflow Syntax & Structure — Easy91 92GitHub Actions YAML has structural problems that GitHub rejects before any job runs.93 94| # | Scenario | What's Broken | Real-World Context |95|---|----------|---------------|-------------------|96| 1 | `checkout_after_build` | `docker build` before `actions/checkout` | No source code — "Dockerfile not found" |97| 2 | `missing_runs_on` | Job has no `runs-on` field | Every job needs a runner |98| 3 | `invalid_trigger_syntax` | `branches: main` instead of `branches: [main]` | Must be a YAML list |99| 4 | `missing_step_uses_or_run` | Step has a name but no `uses:` or `run:` | Invalid step |100| 5 | `missing_on_trigger` | No `on:` block at all | Workflow never triggers |101 102### Task 4: Workflow Secrets & Permissions — Medium103 104Secrets exist but aren't wired correctly to the workflow steps.105 106| # | Scenario | What's Broken | Real-World Context |107|---|----------|---------------|-------------------|108| 1 | `missing_env_secrets` | `$DOCKER_PASSWORD` without `env:` mapping | Secrets must be passed via `env:` block |109| 2 | `wrong_secret_syntax` | `${ secrets.TOKEN }` instead of `${{ secrets.TOKEN }}` | Single vs double braces |110| 3 | `missing_token_permissions` | Pushing to GHCR without `permissions: packages: write` | GITHUB_TOKEN is read-only by default |111| 4 | `secret_not_in_env` | `$SLACK_WEBHOOK_URL` not in `env:` | Very common mistake |112| 5 | `ghcr_wrong_credentials` | Using `DOCKER_PASSWORD` for GHCR login | GHCR uses `GITHUB_TOKEN` |113 114### Task 5: CI + Docker Integration — Medium-Hard115 116The workflow AND the Dockerfile interact. Fixing one file alone isn't enough.117 118| # | Scenario | What's Broken | Real-World Context |119|---|----------|---------------|-------------------|120| 1 | `missing_buildx_for_platforms` | Multi-platform build without `setup-buildx-action` | Need BuildKit for cross-compile |121| 2 | `login_secrets_not_wired` | `docker login` missing `env:` for secrets | "unauthorized: authentication required" |122| 3 | `wrong_build_context` | Context is `./backend` but Dockerfile path is `./Dockerfile` | Path mismatch |123| 4 | `cache_without_mode_max` | GHA cache export missing `mode=max` | Cache doesn't persist |124| 5 | `push_without_login` | `docker push` without `docker login` first | "denied: requested access" |125 126### Task 6: Multi-Stage Pipeline & Matrix — Hard127 128Complex pipelines with multiple interacting bugs. Agent must find 2-3 issues across files.129 130| # | Scenario | What's Broken | Real-World Context |131|---|----------|---------------|-------------------|132| 1 | `artifact_path_mismatch` | `COPY --from=builder /app/dist` but React outputs to `/app/build` | CRA uses `build/`, Vite uses `dist/` |133| 2 | `matrix_platform_arg` | `$BUILDPLATFORM` without `ARG BUILDPLATFORM` | Multi-arch needs platform ARGs |134| 3 | `cross_job_artifact` | Test job downloads artifact but missing `needs: build` | Jobs run in parallel by default |135| 4 | `multiple_issues` | Dockerfile typo + workflow secrets not wired (2 bugs) | Problems compound across files |136| 5 | `matrix_version_failure` | Matrix includes Node 14 but code needs >= 16 + missing `needs:` | 2 bugs to find |137 138### Task 7: Kubernetes Pod Failures — Medium139 140Pod crashes and scheduling failures in Kubernetes deployments.141 142| # | Scenario | What's Broken | Real-World Context |143|---|----------|---------------|-------------------|144| 1 | `oom_killed` | Memory limit 64Mi too low — CrashLoopBackOff/OOMKilled | Most common K8s production issue |145| 2 | `image_pull_backoff` | Image tag typo `nginx:latset` → ImagePullBackOff | Copy-paste tag errors |146| 3 | `wrong_command` | `command: ["python", "workers.py"]` but file is `worker.py` | File name mismatch |147| 4 | `missing_configmap` | `envFrom: configMapRef: app-config` but ConfigMap doesn't exist | CreateContainerConfigError |148| 5 | `liveness_probe_failing` | Liveness probe port 3000 but app listens on 8080 | Probe misconfiguration causes restarts |149 150### Task 8: Kubernetes Service & Ingress Issues — Hard151 152Networking issues where pods run fine but traffic doesn't reach them.153 154| # | Scenario | What's Broken | Real-World Context |155|---|----------|---------------|-------------------|156| 1 | `selector_mismatch` | Service selector `app: api` but pod label is `app: api-server` | No endpoints — most common K8s networking bug |157| 2 | `port_mismatch` | Service targetPort 8080 but container listens on 3000 | Connection refused |158| 3 | `ingress_wrong_service` | Ingress references `api-svc` but service name is `api-service` | Ingress 404 |159| 4 | `network_policy_blocking` | NetworkPolicy with empty ingress rules blocks all traffic | Database unreachable |160| 5 | `missing_ingress_class` | No `ingressClassName: nginx` specified | Ingress controller doesn't pick it up |161 162### Task 9: CI/CD Build & Push Pipeline — Hard163 164GHA-to-Docker-to-Registry pipeline failures spanning multiple files.165 166| # | Scenario | What's Broken | Real-World Context |167|---|----------|---------------|-------------------|168| 1 | `ghcr_token_not_mapped` | `$GITHUB_TOKEN` shell var not mapped from secrets | GHCR login fails |169| 2 | `image_tag_mismatch` | Build uses `github.ref_name` but push uses `github.sha` | "image not found locally" |170| 3 | `missing_packages_write` | No `permissions: packages: write` for GHCR push | "permission_denied: write_package" |171| 4 | `build_arg_not_passed` | Dockerfile `ARG APP_VERSION` but no `--build-arg` in workflow | Version file is empty |172| 5 | `multistage_output_mismatch` | `COPY --from=builder /app/dist` but react-scripts outputs to `/app/build` | Wrong output directory |173 174### Task 10: Full Stack Deployment Pipeline — Expert175 176Multi-error scenarios spanning the entire stack: GHA + Dockerfile + K8s manifests. 2-4 bugs per scenario requiring cross-file reasoning.177 178| # | Scenario | What's Broken | Real-World Context |179|---|----------|---------------|-------------------|180| 1 | `full_pipeline_ghcr_and_selector` | GHCR token not mapped + K8s Service selector mismatch | 2 bugs across workflow + K8s |181| 2 | `full_pipeline_three_bugs` | Missing checkout + no WORKDIR + wrong container/service port | 4 bugs across 4 files |182| 3 | `full_pipeline_ghcr_dockerfile_k8s` | Wrong GHCR secret + base image typo + OOM memory limit | 3 bugs across all layers |183| 4 | `full_pipeline_permissions_image_ingress` | Missing packages:write + hardcoded image placeholder + no ingressClassName | 3 bugs |184| 5 | `full_pipeline_secrets_build_probe` | Docker secrets not wired + wrong build output dir + probe port mismatch | 4 bugs across all layers |185 186---187 188## Available Actions189 190Each step, the agent chooses exactly one action:191 192| Action | What It Does | When to Use |193|--------|-------------|-------------|194| `edit_file` | Replace `old_content` with `new_content` in a file | Most common — fix a broken line or block |195| `replace_line` | Replace content at a specific line number | When you know exactly which line is wrong |196| `add_line` | Insert a new line into a file | Adding missing instructions (e.g., missing `WORKDIR`) |197| `delete_line` | Remove a specific line | Removing a bad instruction |198| `add_block` | Insert a multi-line block | Adding entire sections (e.g., `env:` block with secrets) |199| `delete_block` | Remove a multi-line block | Removing incorrect sections |200| `request_hint` | Get a clue about what's wrong | Costs -5% on final score — use sparingly |201| `submit` | Declare "I'm done" — triggers final evaluation | When all fixes are applied |202 203**Important:** `edit_file` requires `old_content` to match **exactly** (including whitespace). If it doesn't match, the edit fails and the agent gets a -0.02 reward penalty.204 205---206 207## Grading System — How Scores Work208 209Scoring is **deterministic** (same actions always produce the same score) and **dynamic** (different strategies get different scores).210 211### The Formula212 213```214FINAL SCORE = Base + Partial Fixes + Complete Bonus + Efficiency - Hint Penalty - Failed Edit Penalty215```216 217Clamped to `(0.01, 0.99)`.218 219### Component Breakdown220 221| Component | Weight | Description |222|-----------|--------|-------------|223| Base score | 5% | Participation credit |224| Partial fixes | 35% | Proportional to `issues_fixed / issues_total` |225| Complete bonus | 25% | All issues fixed |226| Efficiency | 25% | Decays with extra steps beyond optimal |227| Hint penalty | -4% each | Per `request_hint` action |228| Failed edit penalty | -2% each | Per edit with no valid file path |229 230---231 232## API Endpoints233 234| Endpoint | Method | Description |235|----------|--------|-------------|236| `/` | GET | Root page |237| `/health` | GET | Health check — returns `{"status": "healthy"}` |238| `/metadata` | GET | Environment name, description, version, tags |239| `/schema` | GET | Action, observation, and state JSON schemas |240| `/reset` | POST | Start a new episode (optional: `task_id`, `scenario_id`, `seed`) |241| `/step` | POST | Take an action and receive observation + reward |242| `/state` | GET | Get current observation without taking an action |243| `/info` | GET | Task list with metadata |244| `/tasks` | GET | List all tasks with difficulty levels |245| `/grader` | POST | Grade a trajectory (list of step dicts) |246| `/baseline` | POST | Run built-in heuristic baseline |247| `/mcp` | POST | JSON-RPC 2.0 MCP endpoint (initialize, tools/list) |248 249### Example: Full Episode via API250 251```bash252# 1. Start an episode253curl -X POST http://localhost:8000/reset \254 -H "Content-Type: application/json" \255 -d '{"task_id": "k8s_pod_failures", "scenario_id": "oom_killed"}'256 257# 2. Fix the memory limit258curl -X POST http://localhost:8000/step \259 -H "Content-Type: application/json" \260 -d '{261 "action": {262 "action_type": "edit_file",263 "edits": [{264 "file_path": "k8s/deployment.yaml",265 "old_content": "memory: \"64Mi\"",266 "new_content": "memory: \"256Mi\""267 }]268 }269 }'270 271# Response: reward=0.3, issues_fixed=1/1, done=true272```273 274---275 276## Quick Start277 278### Local Development279 280```bash281pip install -r requirements.txt282python -m uvicorn server.app:app --host 0.0.0.0 --port 8000283```284 285### Run Tests286 287```bash288pytest tests/ -v289```290 291### Docker292 293```bash294docker build -t cloud-native-devops-env .295docker run -p 8000:8000 cloud-native-devops-env296```297 298### Baseline Inference (with LLM)299 300```bash301export API_BASE_URL=https://router.huggingface.co/v1302export MODEL_NAME=meta-llama/Llama-3.1-70B-Instruct303export HF_TOKEN=your_token_here304python inference.py305```306 307---308 309## Project Structure310 311```312cloud-native-devops-env/313├── openenv.yaml # OpenEnv environment specification314├── inference.py # LLM baseline (OpenAI client + HF router)315├── baseline_runner.py # Heuristic baseline for /baseline endpoint316├── Dockerfile # Production container317├── requirements.txt # Python dependencies318│319├── server/320│ ├── app.py # FastAPI with 12 endpoints321│ ├── models.py # Pydantic models (type-safe API)322│ ├── environment.py # Core environment loop (reset/step/state)323│ ├── tasks/324│ │ ├── base.py # BaseTask with scenario loading325│ │ ├── task_registry.py # Maps task_id → task class (10 tasks)326│ │ ├── task_1_build_errors.py # 5 Dockerfile syntax scenarios327│ │ ├── task_2_docker_runtime.py # 5 Dockerfile runtime scenarios328│ │ ├── task_3_workflow_syntax.py # 5 workflow structure scenarios329│ │ ├── task_4_workflow_secrets_permissions.py # 5 secrets scenarios330│ │ ├── task_5_ci_docker_integration.py # 5 integration scenarios331│ │ ├── task_6_multi_stage_matrix.py # 5 multi-issue scenarios332│ │ ├── k8s_pod.py # 5 Kubernetes pod failure scenarios333│ │ ├── k8s_networking.py # 5 K8s networking scenarios334│ │ ├── pipeline_build_deploy.py # 5 GHA→Docker→Registry scenarios335│ │ └── pipeline_full.py # 5 full-stack multi-error scenarios336│ ├── graders/337│ │ └── __init__.py # Deterministic trajectory grader338│ └── simulators/339│ ├── docker_simulator.py # 15+ Dockerfile validation rules340│ ├── workflow_simulator.py # 15+ workflow validation rules341│ └── k8s_simulator.py # Kubernetes manifest validator342│343└── tests/344 ├── test_endpoints.py # API endpoint tests345 ├── test_determinism.py # Grader determinism + score range tests346 ├── test_baseline.py # Heuristic baseline tests347 ├── test_environment_flow.py # Episode flow tests348 └── test_simulators.py # Simulator unit tests349```350 351## Design Decisions352 3531. **Full cloud-native stack**: Docker + GitHub Actions + Kubernetes — the three pillars of modern deployment pipelines.3542. **Simulated validation (no real Docker/K8s)**: Static analysis rules give deterministic results, fast execution, and no security concerns.3553. **Dense rewards**: Partial credit at every step (+0.3 per fix, -0.02 per failed edit) rather than sparse pass/fail.3564. **Difficulty progression**: Easy tasks are single-file, single-issue. Expert tasks are multi-file, multi-issue with interacting bugs across all three layers.3575. **Exact string matching for edits**: Mirrors real file editing — whitespace matters.3586. **50 scenarios from real bugs**: Every scenario is based on actual developer mistakes documented on Stack Overflow, GitHub Issues, and official documentation.359 360## License361 362MIT363