Experimentation
Checkout Redesign A/B Test
- Company
Etsy- Job positions
- Product AnalystData ScientistData Analyst
- Topics
- A/B testingHypothesis testingConfidence intervalsSegment analysisExperiment data quality
The scenario
Etsy: Analyze a four-week checkout experiment, deal with messy tracking, and decide whether the redesign should launch.
Etsy's design team rebuilt the checkout page to cut down on steps, and ran a four-week A/B test in September 2025. You're a product analyst on the checkout team. The designers are keen to launch the redesign to everyone, and the head of product has asked you for a launch decision the team can trust.
Your task
Validate the experiment data, measure the redesign's effect on conversion and order value, and make a clear launch recommendation.
Instructions
- 1Load both files and check the experiment's integrity before measuring anything. Look for users who appear in both groups, orders that don't belong in the analysis, and whether the split between groups matches the planned 50/50.
- 2Decide how to handle each problem you find, apply it consistently, and report how many users and orders you removed.
- 3Use the share of assigned users who placed at least one order after being assigned as the primary metric. Also measure average order value and revenue per user.
- 4Compare the two groups on the primary metric with a confidence interval and a significance test, and state your assumptions.
- 5Break the results down by device and by new vs. returning visitors. Treat these as follow-up checks and account for testing several segments at once.
- 6Estimate what launching would mean for monthly revenue, with a range instead of a single number.
- 7Write a one-page launch recommendation: launch, don't launch, or change and retest, with the evidence behind it.
Datasets
The data is synthetic and does not come from Etsy, but it's modeled on how real companies record it, including the mess. All files come in one download.
assignments.csv
One row per time a user was assigned to a test group, September 2 to 29, 2025.
31,284 rows · 5 columns · 1.4 MB
| Column | Type | Description |
|---|---|---|
| user_id | integer | The user. |
| variant | text | Test group: control (old checkout) or treatment (new checkout). |
| assigned_at | datetime | When the user was assigned (UTC). |
| device | text | Device the user was on when assigned. |
| visitor_type | text | Whether the user had visited the store before: new or returning. |
Preview the first 5 rowsHide preview
| user_id | variant | assigned_at | device | visitor_type |
|---|---|---|---|---|
| 9371 | treatment | 2025-09-02 00:00:06 | desktop | new |
| 14953 | control | 2025-09-02 00:07:55 | mobile | returning |
| 7085 | control | 2025-09-02 00:08:15 | desktop | new |
| 1063 | control | 2025-09-02 00:08:45 | mobile | new |
| 27723 | treatment | 2025-09-02 00:08:59 | mobile | new |
orders.csv
Orders placed by users in the experiment during the test window.
2,670 rows · 4 columns · 101 KB
| Column | Type | Description |
|---|---|---|
| order_id | integer | Unique ID of the order. |
| user_id | integer | The user who placed the order. |
| ordered_at | datetime | When the order was placed (UTC). |
| order_value | decimal | Order total, in USD. |
Preview the first 5 rowsHide preview
| order_id | user_id | ordered_at | order_value |
|---|---|---|---|
| 900001 | 8321 | 2025-08-28 18:45:39 | 46.93 |
| 900002 | 8275 | 2025-08-29 05:07:10 | 34.88 |
| 900003 | 11704 | 2025-08-31 06:14:31 | 64.13 |
| 900004 | 16712 | 2025-08-31 10:35:44 | 56.39 |
| 900005 | 758 | 2025-08-31 19:23:56 | 121.54 |
Hint
Before joining orders, check that each user was only ever in one group.
groups_per_user = assignments.groupby("user_id")["variant"].nunique()
mixed_users = groups_per_user[groups_per_user > 1].indexDeliverable
A public GitHub repo with your analysis (a notebook or scripts) and a README with your data checks, results and launch recommendation.
When you're done, post your repo in the Solutions tab to share it with other learners.
What grading checks
Use this checklist to review your own work before you post and share it.
- Submitted GitHub repo is public and reachable.
- Repo contains at least one notebook or script file.
- Users assigned to both groups and orders placed before assignment are found and handled before results are calculated.
- The group split is checked against the planned 50/50.
- Conversion is compared with a significance test or confidence interval, and assumptions are stated.
- Device breakdown shows the effect is not the same on every device.
- Recommendation follows from the evidence and does not launch a change that hurts a major segment without addressing it.