iac
Datasets
All datasets matching “iac”iac-eval
IaC-Eval dataset (v1.1)
IaC-Eval dataset is the first human-curated and challenging Cloud Infrastructure-as-Code (IaC) dataset tailored to more rigorously benchmark large language models' IaC code generation capabilities.
This dataset contains 458 questions ranging from simple to difficult across various cloud services (targeting AWS for now).
| Github | 🏆 Leaderboard TBD | 📖 NeurIPS 2024 Paper |
2. Usage instructions
Option 1: Running the evaluation… See the full description on the dataset page: https://huggingface.co/datasets/autoiac-project/iac-eval.ia-census
Internet Archive census (April 2016) — per-file md5+sha1
The April 2016 Internet Archive census
recorded md5 and sha1 for every file of every public IA item —
the only bulk file-hash inventory of the Archive ever published (the
census was not repeated; nothing uploaded after 2016-04-11 appears
here). This dataset is those two dumps joined — they are line-aligned
outputs of one item walk — into single witness rows co-observing both
digests, republished as page-indexed parquet for… See the full description on the dataset page: https://huggingface.co/datasets/david-ar/ia-census.iac-terraform-lab
IaC Terraform lab scenarios
Synthetic AWS Terraform stacks for the IaC configuration-review demo: public exposure, a private lookalike, unresolved variables with a remote module, public RDS plus wildcard IAM, and a single-region CloudTrail.
Organization dataset and collection are public. Live Gradio will be alirezaaminzadeh/iac and the organization card AriaAICompany/iac after the daily Space-creation cap resets (scripts/publish.py). Runnable Space source is stored in demo/.… See the full description on the dataset page: https://huggingface.co/datasets/AriaAICompany/iac-terraform-lab.Multi-IaC-Eval
Multi-IaC-Eval
We present Multi-IaC-Eval is a novel benchmark dataset for evaluating LLM-based IaC generation and mutation across AWS CloudFormation, Terraform, and Cloud Development Kit (CDK) formats. The dataset consists of triplets containing initial IaC templates, natural language modification requests, and corresponding updated templates, created through a synthetic data generation pipeline with rigorous validation.
Cloudformation: 263
Terraform: 446
CDK (Python): 64
CDK… See the full description on the dataset page: https://huggingface.co/datasets/AmazonScience/Multi-IaC-Eval.control_arena_iac-monitor-labelsambig-iac-sample
Ambig-IaC Random 50
A random subset of 50 rows sampled without replacement from all 300 rows of
the default/train split of znyang/ambig-iac.
Source revision: 96429693e7164b024333e6d02f8c6b2d017e4ecb.
Sampling: Python random.Random(42).sample(range(300), 50).
Rows are stored in random draw order. All seven original columns, their types,
and original IDs are preserved. Compared with the pinned source, prompt_original
was edited in 10 rows and checks_rego in 2 rows. The other five… See the full description on the dataset page: https://huggingface.co/datasets/shihanlin/ambig-iac-sample.