IaC Eval Bench: Auditing Static Evaluators for LLM-Generated Kubernetes Configurations

Deavyansh Gautam; H Zhang

Evaluating generated infrastructure configuration requires more than checking that an output parses. A configuration can satisfy a schema while violating a task requirement; conversely, a correct configuration can fail an overly restrictive grader. This report presents IaC Eval Bench, a reproducible static evaluation harness for a constrained Kubernetes configuration-generation interface. The suite contains six original synthetic tasks and 50 deterministic requirement checks, with constructed positive and negative controls. An internal scoring audit identified five grading errors and one resume-provenance defect. The grading errors included both false acceptance of contradictory configurations and false rejection of compatible additions. We describe the corrected scoring contract, evidence-preserving execution protocol and failure-analysis procedure. A recorded CPU-only experiment evaluated Qwen2.5-Coder 1.5B and 0.5B, each once on all six tasks. Neither model passed a complete task; their macro rubric scores were 17.13% and 0%, respectively. The smaller model's six responses were valid JSON but failed the required response envelope. Replaying the saved outputs reproduced every score. These observations support an engineering contribution in evaluator auditing and experiment integrity, not a model ranking or a claim of general infrastructure competence. The artifacts expose the prompts, outputs, rubrics, diagnostic edits and operational measurements needed to inspect that distinction.

IaC Eval Bench: Auditing Static Evaluators for LLM-Generated Kubernetes Configurations

Abstract

Citation