@ppelberg See my current recommendation below regarding the mobile talk page volume needed to complete the planned DiscussionTool on Mobile analyses. Let me know if you have any questions.

Method: To determine the scope required for these analyses, I reviewed the average monthly talk page contributors for each wiki and estimated the proportion of those users we might expect to use one of the available DiscussionTool features (around 14.8% of users) in a given month. This percent is based on trends seen on wikis where the Reply and New Discussion tool are available as opt-out on desktop. I also took into consideration the volume of events used in previous AB Tests and the planned scope of each DiscussionTools on Mobile analysis.

1. How many unique people using talk pages each month do we think will be required in order to for the two impact analyses we have planned (T298058 + T298065) to yield significant results in a timely manner?

If we only include the planned 8 partner wikis, that would be about 2480 talk page users per month. An estimated 368 of these users would use at least one of the deployed discussiontool features once in a given month. That should be just sufficient to complete the planned prep and post deployment analyses across all partner wikis.

However, there may not be sufficient data to be able to confidently identify trends on a per wiki basis. For the partner wikis with a smaller number of talk page contributors, such as viwiki, we may only see 19 or fewer distinct talk page users complete an edit with one of the deployed DiscussionTools features in the reviewed timeframe. That will likely not be sufficient to identify any pre and post-deployment changes within that particular wiki.

If possible, I'd recommend adding at least one or two more partner wikis with higher average monthly talk page contributors to the partner list (I'd recommend wikis with 200 or more average monthly mobile talk page users if possible). This would increase the overall amount of data available to review and also provide more wikis that have sufficient data to identify any per wiki trends in usage.

2. How many unique people using talk pages each month do we think will be required in order to for the A/B test we have planned (T298062) to yield significant results in a timely manner?

We would need to make sure to include different wikis for the AB test than the partner wikis used for the T298058 + T298065 analyses. There should ideally be at least 2000 total unique people using talk pages each month across all selected partner wikis.

In addition, I'd recommend expanding the number of partner wikis in this test. In the AB Test, we can account for the effect of the wiki on the success probability of a Junior Contributor completing an edit, which will allow us to more accurately understand the effect of the Mobile DiscussionTools. The more distinct wikis we are able to include in the test, the more data we will have to account for the effect of each wiki. For reference, we used a set of 15 partner wikis in the Reply Tool AB Test, which was sufficient for the analysis.

Smaller wikis can be included but, if we want to make any per wiki conclusions, I'd recommend including at least four or five partner wikis with over 200 unique talk page users each month.

Question: To be doubly sure: does the *2,480* number above include people who are logged out?

@ppelberg No this only accounts for logged-in users. We do not have an accurate count of distinct logged-out users that edit Talk Pages on mobile based on current instrumentation.

If we decide to include logged-out users from those wikis as well, we can look at adding an anon token to identify these users in the test but don't have data to currently confirm the number of estimated logged-out talk page users each month on those wikis.

In T298271#7693761, @MNeisler wrote:

Question: To be doubly sure: does the *2,480* number above include people who are logged out?

@ppelberg No this only accounts for logged-in users. We do not have an accurate count of distinct logged-out users that edit Talk Pages on mobile based on current instrumentation.

Understood.

If we decide to include logged-out users from those wikis as well, we can look at adding an anon token to identify these users in the test but don't have data to currently confirm the number of estimated logged-out talk page users each month on those wikis.

Per the discussion @MNeisler and I had today, it is likely that we'll need to analyze the behavior of logged out users as part of the analyses we have planned.

Although, as Megan noted, no further instrumentation is required to analyze logged out user behavior considering we already implemented this as part of https://gerrit.wikimedia.org/r/c/748157.