AI
HR Policy Assistant with RAG
- Company
Workday- Job positions
- AI EngineerML EngineerData Scientist
- Topics
- RAGEmbeddingsVector searchLLM evaluationHallucination checks
The scenario
Workday: Build a RAG assistant that answers employee questions from policy documents, cites its source and says so when the answer isn't there.
Workday's People team answers the same employee questions every day: how much PTO rolls over, what the hotel limit is, when a raise takes effect. They want an internal assistant that answers from the company's policy documents. Leadership has one condition: it must never make up a policy. You're an AI engineer asked to build it and prove it is trustworthy.
Your task
Build a retrieval-augmented generation (RAG) assistant that answers questions from the policy documents, cites the document it used, declines questions the documents do not cover, and measure how well it does.
Instructions
- 1Embed the policy documents and build retrieval that returns the most relevant documents for a question. Measure how often the correct document is in the top 3 on questions_dev.csv, ignoring the questions whose source is NONE.
- 2Write a prompt that answers only from the retrieved documents, names the doc_id it used, and replies that the documents do not cover the question when they don't.
- 3Evaluate on questions_dev.csv: citation accuracy, whether each answer contains its answer_key_fact, and how many NONE questions the assistant correctly declines.
- 4Try at least one improvement, such as retrieving more documents, adding the section and title to each document before embedding, or a stricter prompt, and report whether it helped.
- 5Answer every question in questions_test.csv and save answers.csv with the columns question_id, answer and source_doc_id. Use NONE as the source_doc_id when the assistant declines.
- 6Write a README with your design, evaluation results, three failures you found and what they would mean for employees relying on the assistant.
Datasets
All files come in one download.
policy_documents.csv
28 short company policy documents.
28 rows · 4 columns · 5 KB
| Column | Type | Description |
|---|---|---|
| doc_id | text | Unique ID of the document. |
| section | text | Handbook section, such as Time off or Travel. |
| title | text | Policy title. |
| text | text | The policy text. |
Preview the first 5 rowsHide preview
| doc_id | section | title | text |
|---|---|---|---|
| D01 | Time off | Paid time off | Full-time employees earn 1.25 days of paid time off each month, which adds up to 15 days a year. Up to 5 unused days carry over into the next year. Any other unused days expire on January 31. |
| D02 | Time off | Requesting time off | Requests for 3 or more days in a row must be made in the time off app at least 2 weeks before the first day. Managers approve or decline requests within 3 business days. |
| D03 | Time off | Sick days | Every employee gets 10 paid sick days a year, separate from paid time off. A note from a doctor is needed after 3 sick days in a row. |
| D04 | Time off | Company holidays | The company observes 11 paid holidays each year. Employees also get 2 floating holidays to use on any day they choose. Floating holidays do not carry over to the next year. |
| D05 | Leave | Parental leave | Employees with at least 6 months of service get 16 weeks of fully paid parental leave for the birth, adoption or foster placement of a child. The leave can be taken within 12 months of the child joining the family, in up to two separate blocks. |
questions_dev.csv
Employee questions with answers, for building and evaluating your assistant. Some questions are not covered by any document.
32 rows · 4 columns · 2 KB
| Column | Type | Description |
|---|---|---|
| question_id | text | Unique ID of the question. |
| question | text | What the employee asked. |
| answer_key_fact | text | A short fact a correct answer must contain. Empty when the documents do not cover the question. |
| source_doc_id | text | The document that answers the question, or NONE. |
Preview the first 5 rowsHide preview
| question_id | question | answer_key_fact | source_doc_id |
|---|---|---|---|
| DEV001 | Which days do hybrid employees go into the office? | Tuesdays and Thursdays | D18 |
| DEV002 | What is the hotel limit in San Francisco? | $350 | D16 |
| DEV003 | Is there a company car program? | empty | NONE |
| DEV004 | What is the daily food allowance on a business trip? | $75 | D14 |
| DEV005 | How long is paid parental leave? | 16 weeks | D05 |
questions_test.csv
Held-out employee questions without answers, including some the documents do not cover.
36 rows · 2 columns · 2 KB
| Column | Type | Description |
|---|---|---|
| question_id | text | Unique ID of the question. |
| question | text | What the employee asked. |
Preview the first 5 rowsHide preview
| question_id | question |
|---|---|
| TST001 | Can I claim mileage for driving to the office every day? |
| TST002 | I just joined 4 months ago. Can I take parental leave? |
| TST003 | How much is the relocation package? |
| TST004 | What should I do with a suspicious email? |
| TST005 | Where do I record my volunteering hours? |
Hints
Use any embedding model and LLM you like. If you want to keep costs at zero, all-MiniLM-L6-v2 from sentence-transformers runs on a laptop CPU, and Ollama runs both nomic-embed-text and small open models such as Llama 3.2 3B. Google AI Studio and Groq also offer free API tiers. Saving model responses to a file means reruns cost nothing.
Retrieval is usually the weak link. Check that the right document is being retrieved before you spend time on the prompt.
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("all-MiniLM-L6-v2")
scores = util.cos_sim(model.encode(question), model.encode(document_texts))Tell the model exactly what to say when the documents do not contain the answer, and check for that phrase in code. A fixed phrase is far easier to evaluate than a free-form apology.
Deliverable
A public GitHub repo with your RAG pipeline, cached model responses or a script that recreates them, answers.csv for the test questions, and a README with your design, evaluation results and failure analysis.
When you're done, post your repo in the Solutions tab to share it with other learners.
What grading checks
Use this checklist to review your own work before you post and share it.
- Submitted GitHub repo is public and reachable.
- Repo contains at least one notebook or script file.
- answers.csv has an answer and a source_doc_id for all 36 test questions.
- The cited source_doc_id is correct for at least 80% of test questions the documents cover.
- The assistant declines at least 6 of the 8 test questions the documents do not cover.
- The README reports retrieval, citation and answer accuracy on the dev questions, before and after an improvement.
- The README explains three failures and their risk for employees.