@ovasileva - Can you confirm the deployment dates for each of the test wikis?

@MNeisler - sure, here you go:

Deployment schedule

I created a draft dashboard in Superset to review the opt in and opt out rates for the new desktop improvement features with filters to review by test wiki and edit count bucket. The data in the dashboard comes from PrefUpdate and uses the event.value field to track if the user selected legacy or modern skin versions and only includes events that occurred after the deployment date for each wiki. Even with these changes, there appear to be several issues with how these events are recorded in PrefUpdate that are causing data to appear off from what is expected. For example, we're seeing a really large number of modern vector selections for users with 0 edits, which might be in part due to T260867. Also, the number of opt-in and opt-out numbers don't seem to match what is expected based on which skin was provided as default.

Instead of using PreUpdate, the team is instead looking into creating a new schema that will only track the opt-ins and opt-outs of desktop improvements in T261842. Once that task is complete, I'll recreate the dashboard and create a report on the desktop improvement feature opt-out rate.

Currently looking into providing current vector skin version selections using the the mediawiki user_properties table pending new instrumentation to track opt-outs.

@ovasileva

Here's are the current nondefault user VectorSkinVersion preferences as of 12 October 2020. Data comes from the mediawiki user_properties table.

Some clarifying notes:

• Data reflects the current state and does not account for users that may have opt'd in and opt'd out multiple times since deployment.
• Data recorded in the user properties tables reflects all current nondefault user preferences. If a user has not made any changes to their vector skin version then there is no skin preference recorded.
• Only accounts for logged-in users.
• Based on the context that the new vector skin was deployed as the default setting for all logged-in users on the wikis below, each of the values below can be interpreted as follows: (I ran a check using my user profile to confirm):
  • Legacy: These are users that have currently opt'd out of the modern version.
  • Modern: This number does not reflect the total number of users that are using the modern skin; only the users that made changes to their default preferences (i.e. These are users that opt'd out and then opt'd back in to the modern version).
  • Unknown: There were a few VectorSkinVersion values set to 0 (instead of 1 [legacy] or 2[modern]). I need to further investigate what those values indicate.
  • Opt-Out Rate: Calculated by dividing the total number of users with legacy preference set by the total number of all registered users on the wiki. It might be useful to look at opt-out rate among just active users as well. I can look into if needed.

Overall

By Target Wiki

[1] These are users that opt'd out and then opt'd back into the modern version. (Note: This number does not reflect the total number of users that are using the modern skin; only the users that made changes to their default preferences).
[2] Calculated by dividing the total number of users with their vector version preference changed to 'legacy' by the total number of all registered users on the wiki.

See notebook for further details.

I reviewed the opt-out rate among active editors (users that had 5 or more content edits overall in the last year from September 2019 to September 2020).

Approach: The opt out rate was calculated by finding the percent of active editors for each wiki (obtained using data from mediawiki_history) that have VectorSkinVersion preference set to legacy in the user properties table.

Assumptions

• Since the modern vector version was deployed as default to all of the test wikis in this analysis, it was assumed that any users with a non-default preference recorded as legacy have opt-d out.
• There are a large number of active editors who do not have data for vector skin preference in the user properties table indicating that they are set to the default 'Modern' skin OR due to being deleted from the user_preference table. In the numbers represented below, all of these users were all added as 'Modern' skin users.

Active Editor Opt-Out Rate Across all Target Wikis

Active Editor Opt-Out Rate For Target Wiki

The opt out rates among active editors for each target wiki are still well below 40%. Persian Wikipedia (fawiki) currently has the highest opt out rate (15.1%) among active editors while Basque Wikipedia (euwiki) has the lowest (5.1%). See notebook for further details.
@ovasileva - Let me know if you have any questions!