Check on Section Editing A/B test results
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	ppelberg
	May 14 2019, 4:58 PM

Description

Overview

We should now have enough data to begin analyzing the impact of the Section Editing A/B test (see: T218851). This task is about doing a quick check to see what impacts the test has had.

"Done"

Answer the following questions:

Has there been a clear change in the number of mobile VE edit sessions?
Has there been a clear change in the edit completion rate in either the treatment or control group?
What – if anything - should we be concerned about?

Generate graphs showing:

Edit completion rate: treatment vs. control
Edit sessions: treatment vs. control
Users: treatment vs. control

References

The last check we did on the Section Editing A/B test: T220460
The output of our last check: Visual section editing experiment sanity check

Related Objects
Search...

Status	Assigned	Task
Open	None	T209955 [EPIC: Focus] Isolate Section Editing
Declined	None	T221327 [Epic] Produce Section Editing + Loading Overlay Report
Duplicate	nshahquinn-wmf	T211239 Analyze impact of section editing in the mobile visual editor
Resolved	mpopov	T223302 Check on Section Editing A/B test results
Resolved	ppelberg	T225645 Turn off section editing A/B test

Event Timeline

ppelberg created this task.May 14 2019, 4:58 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 14 2019, 4:58 PM

• marcella assigned this task to nshahquinn-wmf.May 15 2019, 4:08 PM

• marcella added a project: Product-Analytics.

ppelberg updated the task description. (Show Details)May 16 2019, 1:39 AM

ppelberg updated the task description. (Show Details)

nshahquinn-wmf moved this task from Triage to Next Up on the Product-Analytics board.May 20 2019, 10:35 PM

nshahquinn-wmf added a parent task: T211239: Analyze impact of section editing in the mobile visual editor.May 21 2019, 10:23 PM

ppelberg mentioned this in T224615: Decide next steps for Section Editing (mobile).May 29 2019, 6:38 PM

ppelberg added a subscriber: mpopov.Jun 6 2019, 12:37 PM

nshahquinn-wmf moved this task from Next Up to Icebox on the Product-Analytics board.Jun 10 2019, 11:48 AM

nshahquinn-wmf moved this task from Icebox to Next Up on the Product-Analytics board.

nshahquinn-wmf merged a task: T211239: Analyze impact of section editing in the mobile visual editor.Jun 10 2019, 11:54 AM

nshahquinn-wmf added subscribers: JTannerWMF, nshahquinn-wmf, • DannyH and 4 others.

Mikhail is going to take care of this. 👏👏👏

ppelberg added a subtask: T225645: Turn off section editing A/B test.Jun 12 2019, 7:56 PM

Neil is reviewing my analysis (https://analytics.wikimedia.org/published/notebooks/editing/section-editing-experiment.html)

Okay, here are the daily user counts, VE session counts, sessions/user rates, and edit completion (success) rates:

section-editing-daily.png (1×1 px, 232 KB)

Key takeaways from the analysis:

No clear change in sessions/user
- The average number of sessions per user was 1.94 in the control group and 1.948 in the treatment group; this change was not statistically significant
- We're good to keep section editing on without worrying that it may negatively impact this metric
Clear improvement in completion rate
- There was statistically significant (p-value = 0.034) improvement of 0.98% (in probability of completion) on average
  - For example: on Korean Wikipedia the probability that a started mobile VE session would would end in an edit goes from 36.75% to 37.73% when section editing is enabled
  - The greatest impact is estimated to be on Catalan Wikipedia, where the improvement is 1.042% (from 45.36% in controls to 46.41% in treatment)
  - Refer to https://analytics.wikimedia.org/published/notebooks/editing/section-editing-experiment.html#Per-wiki-improvements for additional examples
- This roughly translates to 1 more completed VE edit for every 100 started if section editing is enabled than if it's not; or, viewed in another way, 1 fewer abandoned VE edit for every 100 started

@ppelberg: My analysis of the data suggests that we should do a full rollout of section editing on mobile VE :D

mpopov moved this task from Next Up to Doing on the Product-Analytics board.Jun 14 2019, 3:24 PM

matmarex subscribed.Jun 14 2019, 5:19 PM

My analysis of the data suggests that we should do a full rollout of section editing on mobile VE :D

This is great news. Thanks for the analysis @mpopov

• marcella awarded a token.Jun 17 2019, 1:39 PM

In T223302#5262679, @iamjessklein wrote:

My analysis of the data suggests that we should do a full rollout of section editing on mobile VE :D

This is great news. Thanks for the analysis @mpopov

+1 – thank you for stepping in and stepping up to bring this all together, @mpopov! We will be turning off the A/B test and rolling out Section Editing to all mobile VE today. See: T225645#5259886

ppelberg awarded a token.Jun 17 2019, 2:18 PM

ppelberg closed subtask T225645: Turn off section editing A/B test as Resolved.Jun 17 2019, 10:06 PM

Thanks @mpopov and @Neil_P._Quinn_WMF! I'm closing this as resolved.

kzimmerman closed this task as Resolved.Jun 18 2019, 2:54 PM

nshahquinn-wmf mentioned this in T226865: Publish Section Editing A/B test results on-wiki.Jul 1 2019, 3:31 PM

nshahquinn-wmf mentioned this in T221327: [Epic] Produce Section Editing + Loading Overlay Report.Jul 12 2019, 12:40 PM