Deterministic Graders (for LLM / AI Evaluation)
Definition A deterministic grader is an evaluation function that produces the same result every time for the same input — no randomness, no LLM-in-the-loop judgment. You check the model’s output against a fixed, code-based rule. Concrete Examples Exact string match — “Does the output equal Paris?” Regex match — “Does the output contain a valid ISO date?” Structured-output validation — “Does this parse as JSON and pass the schema?” Code execution / unit tests — “Run the generated function against these test cases. Did they pass?” Numeric tolerance — “Is the answer within 0.01 of the expected value?” Set membership — “Is the classification label one of {positive, negative, neutral}?” Contrast: Model-Graded / LLM-as-Judge The opposite approach is a model-graded (or “LLM-as-judge”) evaluator, where you ask another model something like “Is this answer helpful and correct?” ...