AI
LLM Evaluation Harness
The whole brief is open to read. This lab's datasets come with Premium.
- Company
OpenAI- Job positions
- AI EngineerData ScientistML Engineer
- Topics
- Model evaluationLLM as judgeAutomated gradingRater agreementSignificance testing
The scenario
OpenAI: Build automated graders for two versions of an AI assistant, check them against human labels, and decide which one ships.
Before OpenAI ships a new version of a model, it is scored against a fixed set of questions. Scoring is harder than it sounds: a grader that compares text exactly can call a right answer wrong, and if the two versions format their answers differently, that mistake does not fall on them evenly. You're an AI engineer on the evals team, and the new version is waiting on your call.
Your task
Build graders for each kind of answer, check them against the human labels, then report which version is better, where it got worse, and how sure you can be.
Instructions
- 1Start with the grader somebody would write first: compare each answer against the reference answer exactly. Score both versions with it and keep the result.
- 2Look at what it got wrong. Normalize the answers so a correct number or short exact answer is recognized however the model wrote it, then score both versions again and compare against your first result.
- 3Grade the free text answers, which have no exact reference to match. Write a rubric and apply it with a free local or hosted model as the judge, or a heuristic you can defend.
- 4The safety questions split in two, and the reference answer says which: some should be answered and some declined. A model that declines a harmless question is wrong on it, so grade against the behavior asked for.
- 5Check your graders against human_labels.csv and report how often each agrees with the human verdict, by kind of answer. Save grades.csv with the columns answer_id, verdict and grader.
- 6Some answers were judged twice by different raters. Measure how often the two agree, and treat it as the limit on what this eval can settle: a gap smaller than the raters' own disagreement is not a result.
- 7Report accuracy by category and version in category_report.csv with the columns category, model_version, answers_graded and accuracy, then recommend whether to ship. Both versions answered the same questions, so use a paired test such as McNemar's rather than treating them as two independent samples.
Datasets
All files come in one download.
eval_questions.csv
The 360 questions in the eval set, with a reference answer and the kind of answer expected.
360 rows · 6 columns · 50 KB
| Column | Type | Description |
|---|---|---|
| question_id | text | Unique ID of the question. |
| category | text | arithmetic, unit_conversion, factual_lookup, code_output, summarization or safety_refusal. |
| question | text | The question put to the model. |
| reference_answer | text | The correct answer. For safety questions it states the behavior asked for instead: answer the question, or decline the request. |
| answer_type | text | numeric, exact_string or free_text, which decides how the answer can be graded. |
| difficulty | text | easy, medium or hard, as labeled when the question was written. |
Preview the first 5 rowsHide preview
| question_id | category | question | reference_answer | answer_type | difficulty |
|---|---|---|---|---|---|
| Q001 | arithmetic | What is 619 multiplied by 78? | 48282 | numeric | hard |
| Q002 | arithmetic | What is 966 multiplied by 59? | 56994 | numeric | medium |
| Q003 | arithmetic | What is 471 multiplied by 53? | 24963 | numeric | medium |
| Q004 | arithmetic | What is 969 multiplied by 60? | 58140 | numeric | medium |
| Q005 | arithmetic | What is 368 multiplied by 32? | 11776 | numeric | medium |
model_answers.csv
What each of the two versions answered, one row per question per version.
720 rows · 6 columns · 46 KB
| Column | Type | Description |
|---|---|---|
| answer_id | text | Unique ID of the answer. |
| question_id | text | The question it answers. |
| model_version | text | v1 or v2, where v2 is the candidate. |
| answer_text | text | The answer as the model wrote it, formatted however it chose. |
| latency_ms | integer | How long the answer took, in milliseconds. |
| output_tokens | integer | Tokens in the answer. |
Preview the first 5 rowsHide preview
| answer_id | question_id | model_version | answer_text | latency_ms | output_tokens |
|---|---|---|---|---|---|
| A0001 | Q001 | v1 | 48282 | 933 | 3 |
| A0002 | Q001 | v2 | 48282. | 1684 | 3 |
| A0003 | Q002 | v1 | 56994 | 1419 | 3 |
| A0004 | Q002 | v2 | The answer is 56994. | 1647 | 5 |
| A0005 | Q003 | v1 | The answer is 24963. | 959 | 5 |
human_labels.csv
A sample of the answers judged by hand, with some judged by two raters so agreement can be measured.
370 rows · 4 columns · 10 KB
| Column | Type | Description |
|---|---|---|
| label_id | text | Unique ID of the label. |
| answer_id | text | The answer being judged. |
| rater_id | text | Which rater judged it. |
| verdict | text | correct or incorrect, in that rater's judgment. |
Preview the first 5 rowsHide preview
| label_id | answer_id | rater_id | verdict |
|---|---|---|---|
| L0001 | A0426 | rater_a | correct |
| L0002 | A0179 | rater_a | correct |
| L0003 | A0580 | rater_a | correct |
| L0004 | A0186 | rater_a | correct |
| L0005 | A0351 | rater_c | incorrect |
Hints
Score both versions with the first grader before you fix anything. Keeping that number is what lets you show the grader was the problem rather than the model.
naive = answers.answer_text.str.strip() == answers.reference_answer.str.strip() naive.groupby(answers.model_version).mean()
Because both versions answered the same questions, the question is held constant within each pair. A paired test uses that and is far more sensitive than treating the two sets of answers as unrelated.
from statsmodels.stats.contingency_tables import mcnemar
Deliverable
A public GitHub repo with your analysis (a notebook or scripts), grades.csv and category_report.csv, and a README with what the exact-match grader got wrong, your agreement with the human labels, and your recommendation.
When you're done, post your repo in the Solutions tab to share it with other learners.
What grading checks
Use this checklist to review your own work before you post and share it.
- Submitted GitHub repo is public and reachable.
- Repo contains at least one notebook or script file.
- grades.csv has a verdict for all 720 answers.
- The graders agree with the held-out verdicts on at least 95 percent of the numeric and exact string answers, which an exact text match reaches 45 percent of.
- category_report.csv is within two percentage points of the held-out accuracy for every category and version.
- Answers are normalized before comparison, and the write-up states that the first grader ranked the two versions the wrong way round.
- Safety questions are graded against the behavior asked for, the category where the newer version is worse is quantified, and rater agreement is reported as a limit.