Review Insights Extraction with an LLM

medium3–4 hours2 datasets

Company: Wayfair
Job positions: AI EngineerData ScientistML Engineer
Topics: LLM promptingStructured outputsFew-shot promptingLLM evaluationPython

The scenario

Wayfair: Extract structured data from unstructured product reviews with an LLM, and evaluate your prompts against labeled examples.

Wayfair's product quality team reads thousands of customer reviews to spot furniture that arrives damaged, ships with missing parts or looks nothing like its photos. Reading them by hand is slow, and simple keyword searches get fooled by reviews like "not a single scratch". You're an AI engineer on the team, asked to use a language model to pull structured facts out of every review.

Your task

Build an LLM pipeline that reads each review and returns three fields as structured output, measure its accuracy on a labeled sample, and run it on every unlabeled review.

Instructions

1Read the labeling rules. issue_type is one of none, damaged_in_shipping, missing_parts, quality_defect, not_as_described or hard_to_assemble. When a review mentions more than one problem, use the first one in that list. A problem still counts if customer service fixed it.
2would_recommend is yes when the reviewer says they would recommend or buy the product again, no when they say they would not or are returning it, and not_stated otherwise. mentions_customer_service is yes when the review mentions contacting support.
3Build a keyword baseline and measure its accuracy for each field on reviews_labeled.csv.
4Write a prompt that returns JSON with exactly the three fields. Validate every response, and retry or repair output that is not valid JSON or uses a value outside the allowed list.
5Compare a zero-shot prompt with a few-shot prompt that includes examples from the labeled reviews, using accuracy per field and a confusion matrix for issue_type. Never include a review in the prompt that you are also scoring.
6Run your best prompt on reviews_to_extract.csv and save extractions.csv with the columns review_id, issue_type, would_recommend and mentions_customer_service. Record the model, settings, run time and any cost in your README.

Datasets

All files come in one download.

reviews_labeled.csv

60 reviews labeled by the quality team, for building and evaluating your prompts.

60 rows · 8 columns · 11 KB

Column	Type	Description
review_id	text	Unique ID of the review.
product_category	text	Furniture, Lighting, Rugs or Bedding.
product_name	text	Type of product reviewed.
rating	integer	Star rating from 1 to 5.
review_text	text	What the customer wrote.
issue_type	text	Main problem in the review, following the labeling rules.
would_recommend	text	yes, no or not_stated.
mentions_customer_service	text	Whether the review mentions contacting support: yes or no.

Preview the first 5 rows

review_id	product_category	product_name	rating	review_text	issue_type	would_recommend	mentions_customer_service
R5002	Furniture	dining table	3	The cat has already claimed it. We have had it for a month now. Nothing was missing from the box.	none	not_stated	no
R5004	Rugs	outdoor rug	2	Fits the space perfectly. Took about a week to arrive. Half the tabletop was chipped before we even took it out of the packaging. The chat agent was helpful and set up a return.	damaged_in_shipping	not_stated	yes
R5018	Rugs	outdoor rug	1	We have had it for a month now. One of the legs was cracked when we opened the box. Support offered a partial refund, which helped. Exactly the style we wanted for the patio.	damaged_in_shipping	not_stated	yes
R5022	Furniture	nightstand	2	Reviews said parts were missing but ours was complete. One of the legs was cracked when we opened the box. The light started flickering after a few days. Would never order this again.	damaged_in_shipping	no	no
R5026	Furniture	TV stand	2	Ordered during the holiday sale. The drawer runners broke the second week.	quality_defect	not_stated	no

reviews_to_extract.csv

340 reviews without labels. Extract the three fields for every one.

340 rows · 5 columns · 51 KB

Column	Type	Description
review_id	text	Unique ID of the review.
product_category	text	Furniture, Lighting, Rugs or Bedding.
product_name	text	Type of product reviewed.
rating	integer	Star rating from 1 to 5.
review_text	text	What the customer wrote.

Preview the first 5 rows

review_id	product_category	product_name	rating	review_text
R5001	Rugs	area rug	4	We get compliments on it all the time. We have had it for a month now. We put it together in about 20 minutes, the instructions were clear. Still deciding whether to keep it.
R5003	Furniture	nightstand	2	We get compliments on it all the time. Nothing was missing from the box. The drawer runners broke the second week. Customer service sent a replacement part within a week. The cat has already claimed it. Save your money and look elsewhere.
R5005	Bedding	weighted blanket	4	Ordered during the holiday sale. The drawer runners broke the second week. It showed up with a long scratch across the top, clearly from the trip here. Looks great in our bedroom. Already ordered a second one for the guest room.
R5006	Bedding	comforter set	5	My husband was skeptical but even he likes it. It does the job.
R5007	Lighting	table lamp	2	We have had it for a month now. Opened it up and the headboard had a big gouge in it. Do not buy this.

Hints

Any LLM works for this project. If you want to keep costs at zero, run an open model on your own computer with Ollama, such as Llama 3.2 3B or Qwen 2.5 3B, or use a free API tier such as Google AI Studio or Groq. Saving every response to a file means reruns cost nothing.

Small local models follow formats better when you ask for JSON only, list the allowed values in the prompt and set the temperature to 0.

response = ollama.chat(
    model="llama3.2:3b",
    messages=[{"role": "user", "content": prompt}],
    format="json",
    options={"temperature": 0},
)

Denials such as "nothing was missing" are where keyword rules fail and a good prompt earns its keep. Add one or two of them to your few-shot examples.

Deliverable

A public GitHub repo with your pipeline code, cached model responses or a script that recreates them, extractions.csv for every unlabeled review, and a README comparing the keyword baseline and each prompt, with the model you used and what it cost to run.

When you're done, post your repo in the Solutions tab to share it with other learners.

What grading checks

Use this checklist to review your own work before you post and share it.

Submitted GitHub repo is public and reachable.
Repo contains at least one notebook or script file.
extractions.csv covers all 340 reviews and uses only the allowed values for each field.
issue_type matches the expected label for at least 80% of reviews.
would_recommend matches the expected label for at least 85% of reviews.
The README compares the keyword baseline, a zero-shot prompt and a few-shot prompt on the labeled reviews.
The pipeline validates model output and records the model, settings and cost.