Etsy · Data Science

Checkout Redesign A/B Test

Kirthi SabariSee their profile and skills
hard

The work

GitHub

  1. Etsy Checkout Redesign A/B Test Analysis

    Author: Senior Product Analyst, Checkout Team
    Date: September 2025
    Dataset Window: September 2, 2025 – September 29, 2025

    Executive Summary & Final Launch Recommendation

    Recommendation: CHANGE & RETEST (PHASED LAUNCH)

    • Desktop Launch: LAUNCH IMMEDIATELY to 100% of Desktop users.
    • Mobile & Tablet Launch: DO NOT LAUNCH. Maintain Mobile/Tablet users on the Control checkout.
    • Next Steps: Conduct mobile usability auditing, investigate mobile form friction / payment integration issues, fix touch UX bugs, and launch a dedicated Mobile A/B retest.

    Summary of Key Findings:

    1. Experiment Data Integrity: Out of 31,000 unique assigned users, 284 users (0.92%) were assigned to both control and treatment groups (cross-device/cookie tracking leakage). A total of 85 pre-assignment orders and 19 mixed-user orders were identified and removed. Post-cleaning, 30,716 clean users (15,339 Control vs. 15,377 Treatment) and 2,566 valid post-assignment orders were analyzed.
    2. No Sample Ratio Mismatch (SRM): The clean user split is 49.94% Control vs 50.06% Treatment (chi2 = 0.047, p = 0.8283), matching the planned 50/50 split.
    3. Overall Aggregated Results (Masked Divergence): At the aggregate level, conversion rate rose from 7.26% to 7.73% (+0.47 pp, +6.47% relative), but this overall change is not statistically significant (p = 0.1180) due to large, opposing effects by device type.
    4. Massive Device Divergence (Key Insight):
      • Desktop Users (+34.3% Rel Lift): Conversion rate surged from 7.54% to 10.13% (+2.59 pp, Z = 5.60, p < 0.0001). Revenue Per User (RPU) jumped from $5.38 to $7.28 (+$1.90 RPU).
      • Mobile Users (-23.0% Rel Drop): Conversion rate collapsed from 7.07% to 5.45% (-1.63 pp, Z = -3.86, p = 0.0001). RPU dropped from $4.95 to $3.94 (-$1.01 RPU).
      • Tablet Users (-22.9% Rel Drop): Conversion rate fell from 6.61% to 5.10% (-1.51 pp, p = 0.1064).
    5. Financial Impact Comparison (Test Cohort Scale: ~30.7k Monthly Users):
      • Full Launch (All Devices): Estimated monthly net gain of +$12,612 (95% CI: [-$1,889, +$27,113]). High variance and CI crosses zero because mobile losses offset desktop gains.
      • Phased Launch (Desktop Only): Estimated monthly net revenue gain of +$28,455 (95% CI: [+$17,509, +$39,401]) — capturing 100% of desktop upside while completely eliminating mobile revenue loss.
  2. import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
import os

# Visual styling
plt.style.use('seaborn-v0_8-whitegrid' if 'seaborn-v0_8-whitegrid' in plt.style.available else 'default')
plt.rcParams['font.sans-serif'] = 'Arial'
plt.rcParams['figure.dpi'] = 150

print('Libraries successfully imported!')

  3. 1. Experiment Data Validation & Integrity Checks

    Before performing statistical analysis, we inspect the datasets for:

    • Users assigned to multiple variants (mixed users).
    • Orders placed before user assignment timestamps.
    • Assignments outside the experiment time window.
    • Sample Ratio Mismatch (SRM) against the 50/50 target split.
  4. ass_path = r'checkout-redesign-ab-test-datasets/checkout-redesign-ab-test/assignments.csv'
ord_path = r'checkout-redesign-ab-test-datasets/checkout-redesign-ab-test/orders.csv'

ass = pd.read_csv(ass_path)
ord_df = pd.read_csv(ord_path)

ass['assigned_at'] = pd.to_datetime(ass['assigned_at'])
ord_df['ordered_at'] = pd.to_datetime(ord_df['ordered_at'])

print(f"Raw Assignments rows: {len(ass):,}")
print(f"Raw Unique users: {ass['user_id'].nunique():,}")
print(f"Raw Orders rows: {len(ord_df):,}")
print(f"Raw Unique buyers: {ord_df['user_id'].nunique():,}")

# 1. Mixed user identification
v_counts = ass.groupby('user_id')['variant'].nunique()
mixed_users = set(v_counts[v_counts > 1].index)
print(f"\n[CHECK 1] Users assigned to BOTH control & treatment: {len(mixed_users):,}")

# 2. Pre-assignment orders
ass_sorted = ass.sort_values(['user_id', 'assigned_at'])
first_ass = ass_sorted.groupby('user_id').first().reset_index()
ord_merged = ord_df.merge(first_ass[['user_id', 'assigned_at', 'variant']], on='user_id', how='left')

pre_ass_orders = ord_merged[ord_merged['ordered_at'] < ord_merged['assigned_at']]
print(f"[CHECK 2] Total orders placed BEFORE assignment: {len(pre_ass_orders):,}")

# Clean Data Filtering
clean_users = ass_sorted[~ass_sorted['user_id'].isin(mixed_users)].groupby('user_id').first().reset_index()
clean_orders = ord_df.merge(clean_users[['user_id', 'variant', 'assigned_at']], on='user_id', how='inner')
valid_orders = clean_orders[clean_orders['ordered_at'] >= clean_orders['assigned_at']].copy()

print(f"\n=== POST-CLEANING SUMMARY ===")
print(f"Clean Assigned Users: {len(clean_users):,}")
print(f"Valid Post-Assignment Orders: {len(valid_orders):,}")
print(f"Removed Users: {len(mixed_users):,}")
print(f"Removed Orders: {len(ord_df) - len(valid_orders):,} (19 mixed-user orders + 85 pre-assignment orders)")
  5. # Sample Ratio Mismatch (SRM) Test
n_c = (clean_users['variant'] == 'control').sum()
n_t = (clean_users['variant'] == 'treatment').sum()
total_n = n_c + n_t

chi2, p_val_srm = stats.chisquare([n_c, n_t], [total_n/2, total_n/2])
print(f"Control Users: {n_c:,} ({n_c/total_n:.2%})")
print(f"Treatment Users: {n_t:,} ({n_t/total_n:.2%})")
print(f"Chi-Square Stat: {chi2:.4f}, p-value: {p_val_srm:.4f}")
if p_val_srm > 0.01:
    print("SUCCESS: No Sample Ratio Mismatch (SRM) detected. Traffic split is consistent with 50/50.")
else:
    print("WARNING: SRM detected!")

  6. 2. Overall Primary & Secondary Metrics Analysis

    • Primary Metric: Conversion Rate (% of assigned users who placed at least 1 order post-assignment).
    • Secondary Metrics: Average Order Value (AOV) per order, Revenue Per User (RPU).
  7. # Aggregate orders per user
user_orders = valid_orders.groupby('user_id').agg(
    n_orders=('order_id', 'count'),
    total_revenue=('order_value', 'sum')
).reset_index()

df = clean_users.merge(user_orders, on='user_id', how='left')
df['n_orders'] = df['n_orders'].fillna(0).astype(int)
df['total_revenue'] = df['total_revenue'].fillna(0.0)
df['converted'] = (df['n_orders'] > 0).astype(int)

# Metric Stats
c_df = df[df['variant'] == 'control']
t_df = df[df['variant'] == 'treatment']
c_ord = valid_orders[valid_orders['variant'] == 'control']
t_ord = valid_orders[valid_orders['variant'] == 'treatment']

# 1. Conversion Rate
cr_c = c_df['converted'].mean()
cr_t = t_df['converted'].mean()
diff_cr = cr_t - cr_c
rel_cr = diff_cr / cr_c
se_cr = np.sqrt(cr_c*(1-cr_c)/len(c_df) + cr_t*(1-cr_t)/len(t_df))
z_cr = diff_cr / se_cr
p_cr = 2 * (1 - stats.norm.cdf(abs(z_cr)))
ci_cr = [diff_cr - 1.96*se_cr, diff_cr + 1.96*se_cr]

# 2. AOV
aov_c = c_ord['order_value'].mean()
aov_t = t_ord['order_value'].mean()
diff_aov = aov_t - aov_c
t_stat_aov, p_aov = stats.ttest_ind(t_ord['order_value'], c_ord['order_value'], equal_var=False)
se_aov = np.sqrt(c_ord['order_value'].var()/len(c_ord) + t_ord['order_value'].var()/len(t_ord))
ci_aov = [diff_aov - 1.96*se_aov, diff_aov + 1.96*se_aov]

# 3. RPU
rpu_c = c_df['total_revenue'].mean()
rpu_t = t_df['total_revenue'].mean()
diff_rpu = rpu_t - rpu_c
t_stat_rpu, p_rpu = stats.ttest_ind(t_df['total_revenue'], c_df['total_revenue'], equal_var=False)
se_rpu = np.sqrt(c_df['total_revenue'].var()/len(c_df) + t_df['total_revenue'].var()/len(t_df))
ci_rpu = [diff_rpu - 1.96*se_rpu, diff_rpu + 1.96*se_rpu]

print('=== OVERALL METRICS COMPARISON ===')
print(f'Conversion Rate: Control = {cr_c:.4%}, Treatment = {cr_t:.4%}')
print(f'  Abs Lift: {diff_cr:+.4%} ({diff_cr*100:+.2f} pp), Rel Lift: {rel_cr:+.2%}')
print(f'  95% CI: [{ci_cr[0]:+.4%}, {ci_cr[1]:+.4%}], Z-stat = {z_cr:.4f}, p-val = {p_cr:.4f}')

print(f'Average Order Value (AOV): Control = ${aov_c:.2f}, Treatment = ${aov_t:.2f}')
print(f'  Abs Lift: ${diff_aov:+.2f}, Rel Lift: {diff_aov/aov_c:+.2%}')
print(f'  95% CI: [${ci_aov[0]:+.2f}, ${ci_aov[1]:+.2f}], t-stat = {t_stat_aov:.4f}, p-val = {p_aov:.4f}')

print(f'Revenue Per User (RPU): Control = ${rpu_c:.2f}, Treatment = ${rpu_t:.2f}')
print(f'  Abs Lift: ${diff_rpu:+.2f}, Rel Lift: {diff_rpu/rpu_c:+.2%}')
print(f'  95% CI: [${ci_rpu[0]:+.2f}, ${ci_rpu[1]:+.2f}], t-stat = {t_stat_rpu:.4f}, p-val = {p_rpu:.4f}')

  8. 3. Segment Breakdown & Multiple Testing Correction

    We analyze conversion rate and revenue performance by device type (Desktop, Mobile, Tablet) and visitor type (New vs. Returning), accounting for multiple testing.

  9. def run_segment_analysis(df_input, valid_orders_input, group_col):
    results = []
    for cat in sorted(df_input[group_col].unique()):
        sub = df_input[df_input[group_col] == cat]
        sub_orders = valid_orders_input[valid_orders_input['user_id'].isin(sub['user_id'])]
        c_sub = sub[sub['variant'] == 'control']
        t_sub = sub[sub['variant'] == 'treatment']
        n_c, n_t = len(c_sub), len(t_sub)
        cr_c, cr_t = c_sub['converted'].mean(), t_sub['converted'].mean()
        diff_cr = cr_t - cr_c
        rel_cr = diff_cr / cr_c if cr_c > 0 else 0
        se_cr = np.sqrt(cr_c*(1-cr_c)/n_c + cr_t*(1-cr_t)/n_t)
        z_cr = diff_cr / se_cr if se_cr > 0 else 0
        p_cr = 2 * (1 - stats.norm.cdf(abs(z_cr)))
        rpu_c, rpu_t = c_sub['total_revenue'].mean(), t_sub['total_revenue'].mean()
        diff_rpu = rpu_t - rpu_c
        results.append({
            'Segment': cat,
            'Control Users': n_c,
            'Treatment Users': n_t,
            'Control Conv': cr_c,
            'Treatment Conv': cr_t,
            'Abs Conv Lift (pp)': diff_cr * 100,
            'Rel Conv Lift (%)': rel_cr * 100,
            'Z-stat': z_cr,
            'p-value': p_cr,
            'Control RPU': rpu_c,
            'Treatment RPU': rpu_t,
            'RPU Diff ($)': diff_rpu
        })
    return pd.DataFrame(results)

dev_results = run_segment_analysis(df, valid_orders, 'device')
dev_results['Bonferroni Sig (alpha=0.0167)'] = dev_results['p-value'] < (0.05 / 3)
print('=== DEVICE SEGMENT BREAKDOWN ===')
print(dev_results[['Segment', 'Control Conv', 'Treatment Conv', 'Abs Conv Lift (pp)', 'Rel Conv Lift (%)', 'p-value', 'Bonferroni Sig (alpha=0.0167)', 'RPU Diff ($)']])
  10. vtype_results = run_segment_analysis(df, valid_orders, 'visitor_type')
print('=== VISITOR TYPE BREAKDOWN ===')
print(vtype_results[['Segment', 'Control Conv', 'Treatment Conv', 'Abs Conv Lift (pp)', 'Rel Conv Lift (%)', 'p-value', 'RPU Diff ($)']])

  11. 4. Financial Impact Modeling & Launch Scenarios

    We model monthly revenue impact ranges (95% CIs) under two strategy options:

    1. Full Launch (All Devices): Deploying redesign everywhere.
    2. Phased Launch (Desktop Only): Deploying redesign to Desktop while keeping Mobile/Tablet on Control.
  12. def model_revenue_scenario(sub_df, name):
    c = sub_df[sub_df['variant'] == 'control']['total_revenue']
    t = sub_df[sub_df['variant'] == 'treatment']['total_revenue']
    n_total = len(c) + len(t)
    diff_rpu = t.mean() - c.mean()
    se_rpu = np.sqrt(c.var()/len(c) + t.var()/len(t))
    ci_rpu = [diff_rpu - 1.96*se_rpu, diff_rpu + 1.96*se_rpu]
    monthly_impact = diff_rpu * n_total
    ci_impact = [ci_rpu[0] * n_total, ci_rpu[1] * n_total]
    print(f'=== SCENARIO: {name} ===')
    print(f'Monthly Users in Cohort: {n_total:,}')
    print(f'RPU Delta: ${diff_rpu:+.2f} (95% CI: [${ci_rpu[0]:+.2f}, ${ci_rpu[1]:+.2f}])')
    print(f'Monthly Revenue Impact (Test Cohort): ${monthly_impact:+,.2f}')
    print(f'95% CI Range: [${ci_impact[0]:+,.2f}, ${ci_impact[1]:+,.2f}]\n')

model_revenue_scenario(df, 'Full Launch (All Devices)')
model_revenue_scenario(df[df['device'] == 'desktop'], 'Phased Launch (Desktop Only)')
model_revenue_scenario(df[df['device'] == 'mobile'], 'Mobile Redesign Loss')

  13. 5. Visual Summary Charts

    Below are the generated visual charts summarizing the experiment findings.

  14. from IPython.display import Image, display
display(Image(filename='charts/conversion_rate_by_device.png'))
display(Image(filename='charts/segment_conversion_matrix.png'))
display(Image(filename='charts/revenue_impact_scenarios.png'))

The brief this answers

Analyze a four-week checkout experiment, deal with messy tracking, and decide whether the redesign should launch.

Try it yourself

Other answers to this brief

The same problem, solved by somebody else.

Discussion

Nothing here yet. Ask how they approached it, or tell them what you would have done differently.