NLP
Help Center Search
- Company
Stripe- Job positions
- AI EngineerML EngineerData Scientist
- Topics
- Information retrievalSearch rankingTF-IDF and BM25Evaluation metricsNLP
The scenario
Stripe: Build a search engine that finds the right help article for questions written in customers' own words, and measure it properly.
Stripe's help center has an article for most questions, but businesses describe problems in their own words, like "money didn't reach my bank", and the current search only matches exact words. You're an engineer on the self-serve support team, asked to build a better search and show that it actually helps.
Your task
Build a search system that ranks help articles for a question, measure it with standard retrieval metrics, and return the top three articles for each held-out question.
Instructions
- 1Read the articles and the training questions. Note where the words people use differ from the words in the articles.
- 2Build a simple baseline that ranks articles by the words they share with the question, and measure recall at 3 and mean reciprocal rank on the training questions.
- 3Improve the ranking without large language models or paid APIs, for example with TF-IDF or BM25, stop words, extra weight for titles, or by using the training questions to learn how people phrase each topic.
- 4Keep your evaluation honest: if you add training questions to your index, use cross-validation so a question is never used to find itself.
- 5Show three questions your system still gets wrong and explain why.
- 6For every question in queries_test.csv, return the three best articles in order and save predictions.csv with the columns query_id, rank_1, rank_2 and rank_3.
- 7Write a README comparing every version of your search with its metrics.
Datasets
All files come in one download.
articles.csv
24 help center articles.
24 rows · 4 columns · 5 KB
| Column | Type | Description |
|---|---|---|
| article_id | text | Unique ID of the article. |
| category | text | Help center section the article belongs to. |
| title | text | Article title. |
| body | text | Article text. |
Preview the first 5 rowsHide preview
| article_id | category | title | body |
|---|---|---|---|
| A101 | Payouts | When will I receive my payout? | Payouts are sent to your bank account on a rolling schedule. Most accounts receive funds two business days after a charge settles. Your first payout can take seven to fourteen days while we verify your business. |
| A102 | Payouts | Why was my payout delayed? | A payout can be delayed by a bank holiday, a change to your bank details, or a review of unusual activity. Delayed payouts show a pending status in the dashboard with the expected arrival date. |
| A103 | Payouts | Change the bank account for payouts | You can replace the bank account that receives payouts from your settings. For security, changing bank details pauses payouts for up to three days and sends a confirmation email. |
| A104 | Payouts | Instant payouts to a debit card | Eligible accounts can send funds to a debit card within minutes for a small fee. Instant payouts are limited by a daily maximum and are not available for every card. |
| A105 | Disputes | Responding to a dispute | When a customer disputes a charge with their bank, you can submit evidence such as receipts, shipping tracking and customer messages. Evidence must be submitted before the deadline shown on the dispute. |
queries_train.csv
Search questions with the article that answered them.
216 rows · 3 columns · 11 KB
| Column | Type | Description |
|---|---|---|
| query_id | text | Unique ID of the question. |
| query | text | What the user typed. |
| relevant_article_id | text | The article that answers the question. |
Preview the first 5 rowsHide preview
| query_id | query | relevant_article_id |
|---|---|---|
| Q1001 | why set trial lentgh to 14 days asap | A121 |
| Q1002 | how do i fight a chargeback? | A105 |
| Q1003 | how do i upload id to keep my account active? | A116 |
| Q1004 | hi, dunning settings for subscriptions | A122 |
| Q1005 | need help: upgrade a subscriber mid month on my account | A120 |
queries_test.csv
Held-out questions, including ways of asking that never appear in the training questions.
96 rows · 2 columns · 5 KB
| Column | Type | Description |
|---|---|---|
| query_id | text | Unique ID of the question. |
| query | text | What the user typed. |
Preview the first 5 rowsHide preview
| query_id | query |
|---|---|
| Q5001 | help extra security code at sign in |
| Q5002 | fee for instant transfer |
| Q5003 | why take payment after shipping |
| Q5004 | refund bounced because card was cancelled |
| Q5005 | why switch pricing tier for a user on my account |
Hints
Recall at 3 counts how often the right article is anywhere in your top three. Mean reciprocal rank also rewards putting it first: a first-place match scores 1, second place 1/2 and third place 1/3.
Stop words and word pairs often help short search questions more than a bigger vocabulary does.
from sklearn.feature_extraction.text import TfidfVectorizer vectorizer = TfidfVectorizer(stop_words="english", ngram_range=(1, 2))
Deliverable
A public GitHub repo with your notebook or scripts, predictions.csv for the test questions, and a README comparing each version of your search and explaining its remaining mistakes.
When you're done, post your repo in the Solutions tab to share it with other learners.
What grading checks
Use this checklist to review your own work before you post and share it.
- Submitted GitHub repo is public and reachable.
- Repo contains at least one notebook or script file.
- Repo contains predictions.csv with query_id and three article IDs for all 96 test questions.
- The right article is in the top three for at least 60% of test questions.
- Improvements are measured against a simple word-matching baseline.
- Evaluation never lets a training question help find itself.
- README explains at least three remaining mistakes.