It's subtly distinct from sampling rate -- we also need to make sure we synchronize the sampling method. If MobileWebUIActions and EditAttemptStep were both sampling at 1/16 that'd still not be helpful if they were sampling a different 1/16.

In T303654#7771288, @DLynch wrote:

It's subtly distinct from sampling rate -- we also need to make sure we synchronize the sampling method. If MobileWebUIActions and EditAttemptStep were both sampling at 1/16 that'd still not be helpful if they were sampling a different 1/16.

Great spot, @DLynch. I've updated the task description to incorporate what you noted above. Please edit the task description if any further adjustment is needed to accurately represent the nuanced you noted in T303654#7771288.

Conveniently: everything about whether or not to log in both desktop and mobile DiscussionTools (assuming nothing triggering oversampling is present), and also the UIActions schemas, goes through mw.eventLog.inSample. Because of how this works, using a global "pageload token" as the sampling ID, any events with the same sampling rate on the same pageload are going to be sampled the same.

*If* we synchronize sampling rate configs, everything should line up. There will be things you can do that will jog them out of alignment, mostly by causing editing stats oversampling. However, if Megan excludes editing sessions with is_oversample from the analysis, it should work. (I think. She might want to comment on whether that'll have any statistical effect.)

(To remind myself when I read this later: This doesn't apply to WikiEditor or desktop VisualEditor. They use the EditAttemptStep session ID as the sampling ID.)

In T303654#7856027, @DLynch wrote:

However, if Megan excludes editing sessions with is_oversample from the analysis, it should work. (I think. She might want to comment on whether that'll have any statistical effect.)

@DLynch would it be accurate for me to think the next step on this task is for @MNeisler to confirm that A) synchronizing sampling rate configs across the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas and B) ensuring that we do NOT oversample any editing-related events will not cause any issues for the analyses we have planned?

@ppelberg Yes, that sounds correct.

@DLynch @ppelberg See responses and follow-up questions below.

@DLynch would it be accurate for me to think the next step on this task is for @MNeisler to confirm that A) synchronizing sampling rate configs across the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas and B) ensuring that we do NOT oversample any editing-related events will not cause any issues for the analyses we have planned?

A) synchronizing sampling rate configs across the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas

The plan described in T303654#7856027 sounds reasonable to me. To clarify, do we anticipate this synchronization to cause any changes to the UIActions schema's current sampling rates? If so, we should keep the Web team in the loop of any changes as it may impact any planning or in-progress analyses using that instrumentation.

FYI current UIActionSchema Sampling Rates (unless changed recently):

• DesktopWebUIActions: 20% on pilot wikis. Ref: T307640#7921880
• MobileWebUIactiosn: 1% for English Wikipedia and 10% for all other wikis. Ref: T295490#7695552

B) ensuring that we do NOT oversample any editing-related events will not cause any issues for the analyses we have planned?

The only issue I see with removing is_oversample events is that it will reduce the number of DiscussionTool sessions available to review within a certain timeframe (In previous analyses, we've included oversampled DiscussionTool events to help increase the sample size available for review). As a result, we may be need to collect data for a longer time period in order to obtain a representative sample, especially for analyses of smaller-sized partner wikis where there are likely to be fewer mobile talk page events. This may impact the timing for T298065 and T298062, which both have metrics that require us joining talk page views from the UIActions schemas to editAttemptStep.

(To remind myself when I read this later: This doesn't apply to WikiEditor or desktop VisualEditor. They use the EditAttemptStep session ID as the sampling ID)

Does this mean that the sampling rates configs WikiEditor or desktop VisualEditor will not be synchronized with the UIActions schemas? In other words, will we only be able to link desktop and mobile DiscussionTool edit events to UIActions schemas?

To clarify, do we anticipate this synchronization to cause any changes to the UIActions schema's current sampling rates?

I don't take a position on that. We'd need to set all their rates to the same value, but it's all the same to me which one is chosen as the base rate. I suspect that's a decision for you and Peter, based on how long it'd take to accumulate enough data to make decisions?

Does this mean that the sampling rates configs WikiEditor or desktop VisualEditor will not be synchronized with the UIActions schemas? In other words, will we only be able to link desktop and mobile DiscussionTool edit events to UIActions schemas?

Yes. Those are both sampled from a different value, and so will be sampled at the same rate but not in sync with the UIActions on that pageload. In the case of WikiEditor we outright can't sync them up without some fairly large rewrites to how the page session token is generated, because the init event happens server-side before that token even exists. (GrowthExperiments has done a little bit in this direction already, I believe, but that's entirely restricted to Special:Homepage and we'd need something more general purpose that'd feed into other existing schemas.)

We could rewrite VE's sampling to also go off of the page token, if we wanted, and it'd be pretty trivial. It might cause a minor difference in the meaning of sampled data, as currently every individual edit session has a chance to be sampled separately, and we'd switch that so that multiple sessions on a pageload would always be sampled together. (But I strongly suspect that most sessions wind up being on their own pageloads.)

Per what @DLynch, @MNeisler, and I talked about last week (18 May):

• 1. Is it feasible to make it so the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas sample events at the same rate, using the same method?

Yes, it is feasible to make it so the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas sample events at the same rate, using the same method.

• 2. If the answer to "1." is "yes," what work would be required to make it so the VisualEditorFeatureUse, EditAttemptStep, MobileWebUIActions, and Talk_Page_Edit schemas sample events at the same rate, using the same method?

To make it so the schemas listed above sample events at the same rate, the Editing Team will need to:

• A) Increase the EditingAttemptStep schema's sampling rate from 6.25% (1 in 16 events) to match the sampling rate of the MobileWebUIActions and DesktopWebUI schemas' sampling rates, both of which are currently set to sample 20% (1 in 5) events.
• B) "Turn on" sampling within the MobileWebUIActions and DesktopWebUI schemas at the Editing Team's nine mobile partner Wikipedias (T294609): ja.wiki, ar.wiki, fr.wiki, ko,wiki, vi.wiki, he.wiki, bn.wiki, zh.wiki, and ht.wiki

Note: work to do A) and B) will happen in T309260.

@ppelberg Yes. None of those require a change to the sampling method, as they don't involve VE/WikiEditor. Those can all be logged via EditAttemptStep or the UIActions schemas.