Stripe · NLP

Help Center Search

Disha AgarwalSee their profile and skills

hard

The work

In [1]:
```
import pandas  as pd
```

In [2]:

articles = pd.read_csv("../Data/articles.csv")
queries_test = pd.read_csv("../Data/queries_test.csv")
queries_train = pd.read_csv("../Data/queries_train.csv")

In [3]:

articles.head(10)

queries_test.head(10)

queries_train.head(10)

  query_id                                              query  \
0    Q1001               why set trial lentgh to 14 days asap   
1    Q1002                       how do i fight a chargeback?   
2    Q1003      how do i upload id to keep my account active?   
3    Q1004             hi, dunning settings for subscriptions   
4    Q1005  need help: upgrade a subscriber mid month on m...   
5    Q1006      need help: how do i fight a chargeback please   
6    Q1007                  hi, cost of loisng a dispute asap   
7    Q1008                        help verify event signature   
8    Q1009                 question: switch from test to live   
9    Q1010       help block suspicious payments on my account   

  relevant_article_id  
0                A121  
1                A105  
2                A116  
3                A122  
4                A120  
5                A105  
6                A106  
7                A124  
8                A123  
9                A107

In [4]:
```
articles.shape
```
```
(24, 4)
```
In [5]:
```
queries_train.shape
```
```
(216, 3)
```
In [6]:
```
queries_test.shape
```
```
(96, 2)
```

In [7]:

articles.isnull().sum()
queries_train.isnull().sum()
queries_test.isnull().sum()

query_id    0
query       0
dtype: int64

#for baseline.py
In [8]:
```
import sys
```

In [9]:

sys.path.append("../src")
from baseline import tokenize

In [10]:

query = queries_train.iloc[0]["query"]
print(query)

why set trial lentgh to 14 days asap

In [11]:

print(tokenize(query))

{'asap', 'why', 'to', 'set', 'days', 'trial', 'lentgh'}

In [12]:

import importlib
import baseline

importlib.reload(baseline)

<module 'baseline' from 'c:\\Users\\Disha Agarwal\\OneDrive\\Desktop\\HelpRank\\Notebooks\\../src\\baseline.py'>

In [13]:

from baseline import word_match_score

query = queries_train.iloc[0]["query"]
article =  articles.iloc[0] 

article_text = article["title"]+""+article["body"]
score = word_match_score(query , article_text)

print (query)
print(article["title"])
print(score)

why set trial lentgh to 14 days asap
When will I receive my payout?
2

In [14]:

import importlib
import baseline

importlib.reload(baseline)

<module 'baseline' from 'c:\\Users\\Disha Agarwal\\OneDrive\\Desktop\\HelpRank\\Notebooks\\../src\\baseline.py'>

In [15]:

from baseline import rank_articles

query = queries_train.iloc[0]["query"]

results = rank_articles(query, articles)

print("Query:", query)
print("\nTop 3 Articles:")

for result in results[:3]:
    print(result)

Query: why set trial lentgh to 14 days asap

Top 3 Articles:
{'article_id': 'A101', 'score': 2}
{'article_id': 'A102', 'score': 2}
{'article_id': 'A103', 'score': 2}

In [16]:

query_row = queries_train.iloc[0]

print("Query:", query_row["query"])
print("Correct Article:", query_row["relevant_article_id"])

print("\nOur Top 3:")
for result in results[:3]:
    print(result)

Query: why set trial lentgh to 14 days asap
Correct Article: A121

Our Top 3:
{'article_id': 'A101', 'score': 2}
{'article_id': 'A102', 'score': 2}
{'article_id': 'A103', 'score': 2}

In [17]:

query_row["relevant_article_id"] in [r["article_id"] for r in results[:3]]

False

In [18]:

def evaluate_baseline(queries, articles):
    recall_at_3 = 0
    reciprocal_ranks = []

    for _, row in queries.iterrows():
        query = row["query"]
        correct_article = row["relevant_article_id"]

        results = rank_articles(query, articles)

        top_3 = [result["article_id"] for result in results[:3]]

        # Recall@3
        if correct_article in top_3:
            recall_at_3 += 1

        # MRR
        if correct_article in top_3:
            rank = top_3.index(correct_article) + 1
            reciprocal_ranks.append(1 / rank)
        else:
            reciprocal_ranks.append(0)

    recall_at_3 = recall_at_3 / len(queries)
    mrr = sum(reciprocal_ranks) / len(reciprocal_ranks)

    return recall_at_3, mrr

In [19]:

recall_at_3, mrr = evaluate_baseline(
    queries_train,
    articles
)

print(f"Baseline Recall@3: {recall_at_3:.4f}")
print(f"Baseline MRR: {mrr:.4f}")

Baseline Recall@3: 0.5648
Baseline MRR: 0.4321

Baseline Results
Method Recall@3 MRR
Word Matching Baseline 56.48% 0.4321
TF-IDF

In [20]:

from tfidf_search import build_tfidf_index

vectorizer, article_vectors = build_tfidf_index(articles)

print("Article vectors shape:", article_vectors.shape)

Article vectors shape: (24, 657)

In [21]:

from tfidf_search import rank_tfidf

query = queries_train.iloc[0]["query"]

results = rank_tfidf(
    query,
    vectorizer,
    article_vectors,
    articles
)

print("Query:", query)
print("\nTop 3 Articles:")

for result in results[:3]:
    print(result)

Query: why set trial lentgh to 14 days asap

Top 3 Articles:
{'article_id': 'A121', 'score': np.float64(0.345439583599322)}
{'article_id': 'A101', 'score': np.float64(0.12160594548943633)}
{'article_id': 'A112', 'score': np.float64(0.0775246292063744)}

In [22]:

def evaluate_tfidf(queries, articles, vectorizer, article_vectors):
    recall_at_3 = 0
    reciprocal_ranks = []

    for _, row in queries.iterrows():
        query = row["query"]
        correct_article = row["relevant_article_id"]

        results = rank_tfidf(
            query,
            vectorizer,
            article_vectors,
            articles
        )

        top_3 = [result["article_id"] for result in results[:3]]

        if correct_article in top_3:
            recall_at_3 += 1

            rank = top_3.index(correct_article) + 1
            reciprocal_ranks.append(1 / rank)

        else:
            reciprocal_ranks.append(0)

    recall_at_3 = recall_at_3 / len(queries)
    mrr = sum(reciprocal_ranks) / len(reciprocal_ranks)

    return recall_at_3, mrr

In [23]:

tfidf_recall, tfidf_mrr = evaluate_tfidf(
    queries_train,
    articles,
    vectorizer,
    article_vectors
)

print(f"TF-IDF Recall@3: {tfidf_recall:.4f}")
print(f"TF-IDF MRR: {tfidf_mrr:.4f}")

TF-IDF Recall@3: 0.7130
TF-IDF MRR: 0.6080

bm_25

In [24]:

%pip install rank-bm25

Collecting rank-bm25
  Using cached rank_bm25-0.2.2-py3-none-any.whl.metadata (3.2 kB)
Requirement already satisfied: numpy in c:\users\disha agarwal\anaconda3\lib\site-packages (from rank-bm25) (2.2.6)
Using cached rank_bm25-0.2.2-py3-none-any.whl (8.6 kB)
Installing collected packages: rank-bm25
Successfully installed rank-bm25-0.2.2
Note: you may need to restart the kernel to use updated packages.

In [25]:

from bm25_search import build_bm25_index

bm25 = build_bm25_index(articles)

print("BM25 index created successfully!")

BM25 index created successfully!

In [26]:

import pandas as pd

articles = pd.read_csv("../Data/articles.csv")
queries_train = pd.read_csv("../Data/queries_train.csv")
queries_test = pd.read_csv("../Data/queries_test.csv")

In [27]:

import sys
import importlib

sys.path.append("../src")

import bm25_search

importlib.reload(bm25_search)

print(dir(bm25_search))

['BM25Okapi', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'build_bm25_index', 'rank_bm25', 're', 'tokenize']

In [28]:

bm25 = bm25_search.build_bm25_index(articles)

query = queries_train.iloc[0]["query"]

results = bm25_search.rank_bm25(
    query,
    bm25,
    articles
)

print("Query:", query)

for result in results[:3]:
    print(result)

Query: why set trial lentgh to 14 days asap
{'article_id': 'A121', 'score': np.float64(5.146791688085988)}
{'article_id': 'A108', 'score': np.float64(1.7488502914746173)}
{'article_id': 'A102', 'score': np.float64(1.7066221697420685)}

In [29]:
```
import bm25_search
```

In [30]:

def evaluate_bm25(queries, articles, bm25):
    recall_at_3 = 0
    reciprocal_ranks = []

    for _, row in queries.iterrows():
        query = row["query"]
        correct_article = row["relevant_article_id"]

        results = bm25_search.rank_bm25(
            query,
            bm25,
            articles
        )

        top_3 = [
            result["article_id"]
            for result in results[:3]
        ]

        if correct_article in top_3:
            recall_at_3 += 1

            rank = top_3.index(correct_article) + 1
            reciprocal_ranks.append(1 / rank)
        else:
            reciprocal_ranks.append(0)

    recall_at_3 = recall_at_3 / len(queries)
    mrr = sum(reciprocal_ranks) / len(reciprocal_ranks)

    return recall_at_3, mrr

In [31]:

bm25_recall, bm25_mrr = evaluate_bm25(
    queries_train,
    articles,
    bm25
)

print(f"BM25 Recall@3: {bm25_recall:.4f}")
print(f"BM25 MRR: {bm25_mrr:.4f}")

BM25 Recall@3: 0.7500
BM25 MRR: 0.6505

In [32]:

import bm25_search
import importlib

importlib.reload(bm25_search)

<module 'bm25_search' from 'c:\\Users\\Disha Agarwal\\OneDrive\\Desktop\\HelpRank\\Notebooks\\../src\\bm25_search.py'>

In [33]:

query = queries_train.iloc[0]["query"]

results = bm25_search.rank_bm25_title_weighted(
    query,
    articles
)

print("Query:", query)
print("\nTop 3 Articles:")

for result in results[:3]:
    print(result)

Query: why set trial lentgh to 14 days asap

Top 3 Articles:
{'article_id': 'A106', 'score': np.float64(0.0)}
{'article_id': 'A107', 'score': np.float64(0.0)}
{'article_id': 'A110', 'score': np.float64(0.0)}

In [34]:

import bm25_search
import importlib

importlib.reload(bm25_search)

<module 'bm25_search' from 'c:\\Users\\Disha Agarwal\\OneDrive\\Desktop\\HelpRank\\Notebooks\\../src\\bm25_search.py'>

In [35]:

title_bm25, body_bm25 = bm25_search.build_title_weighted_bm25(
    articles
)

print("Title-weighted BM25 index created!")

Title-weighted BM25 index created!

In [36]:

query = queries_train.iloc[0]["query"]

results = bm25_search.rank_bm25_title_weighted(
    query,
    title_bm25,
    body_bm25,
    articles
)

print("Query:", query)
print("\nTop 3 Articles:")

for result in results[:3]:
    print(result)

Query: why set trial lentgh to 14 days asap

Top 3 Articles:
{'article_id': 'A121', 'score': np.float64(4.994452675336603)}
{'article_id': 'A105', 'score': np.float64(4.643673852247172)}
{'article_id': 'A104', 'score': np.float64(4.001173237280832)}

In [37]:

def evaluate_title_weighted_bm25(
    queries,
    articles,
    title_bm25,
    body_bm25
):
    recall_at_3 = 0
    reciprocal_ranks = []

    for _, row in queries.iterrows():
        query = row["query"]
        correct_article = row["relevant_article_id"]

        results = bm25_search.rank_bm25_title_weighted(
            query,
            title_bm25,
            body_bm25,
            articles
        )

        top_3 = [
            result["article_id"]
            for result in results[:3]
        ]

        if correct_article in top_3:
            recall_at_3 += 1

            rank = top_3.index(correct_article) + 1
            reciprocal_ranks.append(1 / rank)
        else:
            reciprocal_ranks.append(0)

    recall_at_3 = recall_at_3 / len(queries)
    mrr = sum(reciprocal_ranks) / len(reciprocal_ranks)

    return recall_at_3, mrr

In [38]:

title_recall, title_mrr = evaluate_title_weighted_bm25(
    queries_train,
    articles,
    title_bm25,
    body_bm25
)

print(f"Title-weighted BM25 Recall@3: {title_recall:.4f}")
print(f"Title-weighted BM25 MRR: {title_mrr:.4f}")

Title-weighted BM25 Recall@3: 0.6250
Title-weighted BM25 MRR: 0.4992

In [39]:

import bm25_search
import importlib

importlib.reload(bm25_search)

<module 'bm25_search' from 'c:\\Users\\Disha Agarwal\\OneDrive\\Desktop\\HelpRank\\Notebooks\\../src\\bm25_search.py'>

In [40]:

expanded_articles = bm25_search.build_query_expanded_articles(
    articles,
    queries_train
)

print(expanded_articles.shape)

(24, 4)

In [41]:

print(
    expanded_articles[
        expanded_articles["article_id"] == "A121"
    ]["body"].iloc[0]
)

Subscriptions can start with a free trial of any length. The customer is charged at the end of the trial unless they cancel, and a reminder email can be sent a few days before. why set trial lentgh to 14 days asap question: set trial length to 14 days please customer charged when trial ended please remind users before trial expires for my shop help customer charegd when trial ended question: remind users before trial expires why customer charged when trial ended? can you tell me remind users before trial expires on my account set trial length to 14 days

In [42]:

from sklearn.model_selection import KFold

In [43]:

kf = KFold(
    n_splits=5,
    shuffle=True,
    random_state=42
)

print("5-fold cross-validation ready!")

5-fold cross-validation ready!

In [44]:

train_indices, val_indices = next(
    kf.split(queries_train)
)

train_queries_fold = queries_train.iloc[train_indices]
val_queries_fold = queries_train.iloc[val_indices]

print("Expansion queries:", len(train_queries_fold))
print("Validation queries:", len(val_queries_fold))

Expansion queries: 172
Validation queries: 44

In [45]:

expanded_articles_fold = bm25_search.build_query_expanded_articles(
    articles,
    train_queries_fold
)

print("Expanded articles:", expanded_articles_fold.shape)

Expanded articles: (24, 4)

In [46]:

fold_bm25 = bm25_search.build_bm25_index(
    expanded_articles_fold
)

print("Fold BM25 index created!")

Fold BM25 index created!

In [47]:

fold_results = []

for fold, (train_indices, val_indices) in enumerate(
    kf.split(queries_train),
    start=1
):
    train_queries_fold = queries_train.iloc[train_indices]
    val_queries_fold = queries_train.iloc[val_indices]

    expanded_articles_fold = bm25_search.build_query_expanded_articles(
        articles,
        train_queries_fold
    )

    fold_bm25 = bm25_search.build_bm25_index(
        expanded_articles_fold
    )

    recall, mrr = evaluate_fold(
        val_queries_fold,
        expanded_articles_fold,
        fold_bm25
    )

    fold_results.append({
        "fold": fold,
        "recall_at_3": recall,
        "mrr": mrr
    })

    print(
        f"Fold {fold}: "
        f"Recall@3={recall:.4f}, "
        f"MRR={mrr:.4f}"
    )

Fold 1: Recall@3=1.0000, MRR=1.0000
Fold 2: Recall@3=1.0000, MRR=1.0000
Fold 3: Recall@3=0.9535, MRR=0.9535
Fold 4: Recall@3=0.9767, MRR=0.9535
Fold 5: Recall@3=1.0000, MRR=1.0000

In [48]:

import pandas as pd

fold_results_df = pd.DataFrame(fold_results)

print("\nCross-Validation Results:")
print(fold_results_df)

print("\nAverage Recall@3:",
      fold_results_df["recall_at_3"].mean())

print("Average MRR:",
      fold_results_df["mrr"].mean())


Cross-Validation Results:
   fold  recall_at_3       mrr
0     1     1.000000  1.000000
1     2     1.000000  1.000000
2     3     0.953488  0.953488
3     4     0.976744  0.953488
4     5     1.000000  1.000000

Average Recall@3: 0.9860465116279069
Average MRR: 0.9813953488372092

In [49]:

final_expanded_articles = bm25_search.build_query_expanded_articles(
    articles,
    queries_train
)

print("Final expanded articles:", final_expanded_articles.shape)

Final expanded articles: (24, 4)

In [50]:

final_bm25 = bm25_search.build_bm25_index(
    final_expanded_articles
)

print("Final BM25 index created!")

Final BM25 index created!

In [51]:

predictions = []

for _, row in queries_test.iterrows():
    query_id = row["query_id"]
    query = row["query"]

    results = bm25_search.rank_bm25(
        query,
        final_bm25,
        final_expanded_articles
    )

    top_3 = [
        result["article_id"]
        for result in results[:3]
    ]

    predictions.append({
        "query_id": query_id,
        "rank_1": top_3[0],
        "rank_2": top_3[1],
        "rank_3": top_3[2]
    })

print("Predictions generated:", len(predictions))

Predictions generated: 96

In [52]:

predictions_df = pd.DataFrame(predictions)

predictions_df.to_csv(
    "../predictions.csv",
    index=False
)

print("predictions.csv created successfully!")

predictions.csv created successfully!

In [53]:

print(predictions_df.head())
print("\nShape:", predictions_df.shape)
print("\nColumns:", predictions_df.columns.tolist())

  query_id rank_1 rank_2 rank_3
0    Q5001   A118   A108   A109
1    Q5002   A104   A102   A106
2    Q5003   A114   A101   A122
3    Q5004   A113   A112   A108
4    Q5005   A114   A103   A120

Shape: (96, 4)

Columns: ['query_id', 'rank_1', 'rank_2', 'rank_3']

In [54]:

cv_predictions = []

for fold, (train_indices, val_indices) in enumerate(
    kf.split(queries_train),
    start=1
):
    train_queries_fold = queries_train.iloc[train_indices]
    val_queries_fold = queries_train.iloc[val_indices]

    expanded_articles_fold = bm25_search.build_query_expanded_articles(
        articles,
        train_queries_fold
    )

    fold_bm25 = bm25_search.build_bm25_index(
        expanded_articles_fold
    )

    for _, row in val_queries_fold.iterrows():
        query_id = row["query_id"]
        query = row["query"]
        correct_article = row["relevant_article_id"]

        results = bm25_search.rank_bm25(
            query,
            fold_bm25,
            expanded_articles_fold
        )

        top_3 = [
            result["article_id"]
            for result in results[:3]
        ]

        cv_predictions.append({
            "fold": fold,
            "query_id": query_id,
            "query": query,
            "correct_article": correct_article,
            "rank_1": top_3[0],
            "rank_2": top_3[1],
            "rank_3": top_3[2]
        })

print("CV predictions:", len(cv_predictions))

CV predictions: 216

In [55]:

cv_predictions_df = pd.DataFrame(cv_predictions)

errors = cv_predictions_df[
    ~(
        (cv_predictions_df["correct_article"] == cv_predictions_df["rank_1"]) |
        (cv_predictions_df["correct_article"] == cv_predictions_df["rank_2"]) |
        (cv_predictions_df["correct_article"] == cv_predictions_df["rank_3"])
    )
]

print("Total errors:", len(errors))

Total errors: 3

In [56]:

errors[
    [
        "query_id",
        "query",
        "correct_article",
        "rank_1",
        "rank_2",
        "rank_3"
    ]
].to_string(index=False)

'query_id                                            query correct_article rank_1 rank_2 rank_3\n   Q1091             delete my account permanently please            A119   A101   A118   A107\n   Q1116 help delete my account permanently on my account            A119   A107   A101   A118\n   Q1044                       why clsoe business acocunt            A119   A116   A101   A112'

HelpRank — NLP-Based Help Center Search Engine

HelpRank is an information-retrieval system that ranks help-center
articles based on customer queries written in natural language.

The project compares multiple retrieval approaches and evaluates
them using Recall@3 and Mean Reciprocal Rank (MRR).

Problem Statement

The goal is to build a search engine that can identify the most
relevant help-center article when customers describe their problem
in their own words.

The dataset contains:

24 help-center articles
216 labeled training queries
96 held-out test queries

Each training query is associated with its relevant article.

Project Workflow

Customer Query
↓
Text Processing
↓
Retrieval Model
↓
Rank All Articles
↓
Top-3 Results

Approaches Evaluated

1. Word Matching Baseline

A simple lexical baseline was implemented by counting the number
of unique words shared between the query and each article.

This provides a benchmark for evaluating more advanced retrieval
methods.

2. TF-IDF

TF-IDF was implemented using:

English stop-word removal
Unigrams
Bigrams
Cosine similarity

This reduces the importance of common words and gives more weight
to informative terms.

3. BM25

BM25 was evaluated as an information-retrieval ranking algorithm.

It considers term frequency and document length when calculating
relevance scores.

4. Title Weighting

A title-weighted BM25 approach was tested to determine whether
giving additional importance to article titles improved retrieval.

This approach was rejected because it reduced performance on the
training evaluation.

5. Query Expansion

Customer training queries were incorporated into the searchable
representation of their corresponding articles.

This allows the retrieval system to learn customer-specific
vocabulary that may differ from the wording used in the original
article.

A 5-fold cross-validation strategy was used to prevent a validation
query from being directly included in its own article representation.

Results

Method	Recall@3	MRR
Word Matching Baseline	56.48%	0.4321
TF-IDF + n-grams	71.30%	0.6080
BM25	75.00%	0.6505
BM25 + Title Weighting	62.50%	0.4992
Query Expansion + BM25	98.60%*	0.9814*

\* 5-fold cross-validation performance on the labeled training
queries. The held-out test labels are not available for calculating
Recall@3/MRR locally.

Error Analysis

The query-expanded BM25 model still produced a small number of
retrieval errors.

Account deletion queries

Queries such as:

"delete my account permanently please"
"help delete my account permanently"

failed to retrieve the correct article within the top 3 results.

This indicates that the article representation did not fully
capture the vocabulary used by these customer queries.

Spelling errors

A query such as:

"why clsoe business acocunt"

contains spelling errors. BM25 is primarily lexical and therefore
does not naturally handle misspellings.

Semantic limitations

These errors demonstrate a limitation of lexical retrieval:
BM25 relies heavily on term overlap and does not fully understand
semantic equivalence.

A future version could use dense embeddings or a hybrid lexical +
semantic retrieval system to improve these cases.

Test Predictions

The final model was trained using all 216 labeled training queries.

It was then used to generate top-3 predictions for all 96 held-out
test queries.

The predictions are stored in:

predictions.csv

The file contains:

query_id
rank_1
rank_2
rank_3

Project Structure

HelpRank/
│
├── Data/
│   ├── articles.csv
│   ├── queries_train.csv
│   └── queries_test.csv
│
├── Notebooks/
│   └── search_engine.ipynb
│
├── src/
│   ├── baseline.py
│   ├── tfidf_search.py
│   ├── bm25_search.py
│   └── evaluate.py
│
├── predictions.csv
├── README.md
├── requirements.txt
└── .gitignore

The brief this answers

Build a search engine that finds the right help article for questions written in customers' own words, and measure it properly.

Try it yourself

Discussion

Nothing here yet. Ask how they approached it, or tell them what you would have done differently.

Add to the discussion