[go: up one dir, main page]

Page MenuHomePhabricator

Sticky header edit icons do not show if an A/B test other than a sticky header A/B test is defined
Closed, ResolvedPublic3 Estimated Story PointsBUG REPORT

Description

NOTE: This is a blocker for deploying the sticky header edit icons to production.

List of steps to reproduce (step by step, including full links if applicable):

  • Use the default definition of VectorWebABTestEnrollment (disabled skin-vector-toc-experiment):
  • Enable the sticky header edit icons
$wgVectorStickyHeader = [
        "logged_in" =>  true,
        "logged_out" => false,
];
$wgVectorStickyHeaderEdit = [
        "logged_in" => true,
        "logged_out" => true,
];
  • Log in
  • Scroll to sticky header

What happens?:
The edit icons do not appear in the sticky header.

What should have happened instead?:
The edit icons should show.

Developer notes

I've written a unit test to capture this problem and attached this to the ticket.

QA steps

NOTE: testing use cases might be easier in a local setup with various config configurations

Test the following use cases to verify seeing or not seeing the edit icons in the sticky header:

  • When sticky header is enabled and an A/B test that is neither of the sticky header experiments is enabled (use following config), you should see edit icons in sticky header.
$wgVectorStickyHeader = [
	"logged_in" =>  true,
	"logged_out" => false,
];
$wgVectorWebABTestEnrollment = [
	'name' => 'skin-vector-toc-experiment',
	'enabled' => true,
	'buckets' => [
		'unsampled' => [
			'samplingRate' => 0,
		],
		'control' => [
			'samplingRate' => 0.5,
		],
		'treatment' => [
			'samplingRate' => 0.5,
		]
	]
];
  • When sticky header is enabled and a sticky header A/B test is disabled, you should see edit icons in sticky header
$wgVectorStickyHeader = [
	"logged_in" =>  true,
	"logged_out" => false,
];
$wgVectorWebABTestEnrollment = [
	"name" => "vector.sticky_header_edit",
	"enabled" => false,
	"buckets" => [
		"control" => [
			"samplingRate" => 0
		],
		"unsampled" => [
			"samplingRate" => 0
		],
		"stickyHeaderDisabled" => [
			"samplingRate" => 0
		],
		"stickyHeaderTreatment" => [
			"samplingRate" => 1
		]
	],
];
  • When sticky header is enabled and a sticky header A/B test is enabled and user is in the treatment bucket, you should see edit icons in sticky header
$wgVectorStickyHeader = [
	"logged_in" =>  true,
	"logged_out" => false,
];
$wgVectorWebABTestEnrollment = [
	"name" => "vector.sticky_header_edit",
	"enabled" => true,
	"buckets" => [
		"control" => [
			"samplingRate" => 0
		],
		"unsampled" => [
			"samplingRate" => 0
		],
		"stickyHeaderDisabled" => [
			"samplingRate" => 0
		],
		"stickyHeaderTreatment" => [
			"samplingRate" => 1
		]
	],
];
  • When sticky header is enabled and a sticky header A/B test is enabled and user is NOT in the treatment bucket, you should NOT see edit icons in sticky header
$wgVectorStickyHeader = [
	"logged_in" =>  true,
	"logged_out" => false,
];
$wgVectorWebABTestEnrollment = [
	"name" => "vector.sticky_header_edit",
	"enabled" => true,
	"buckets" => [
		"control" => [
			"samplingRate" => 0
		],
		"unsampled" => [
			"samplingRate" => 0
		],
		"stickyHeaderDisabled" => [
			"samplingRate" => 1
		],
		"stickyHeaderTreatment" => [
			"samplingRate" => 0
		]
	],
];

QA Results - Beta

ACStatusDetails
1T310750#8054160
2T310750#8054160

QA Results - Prod

ACStatusDetails
1T310750#8069513
2T310750#8069513

Event Timeline

Change 805911 had a related patch set uploaded (by Jdlrobson; author: Jdlrobson):

[mediawiki/skins/Vector@master] Test case for T310750

https://gerrit.wikimedia.org/r/805911

Change 805911 abandoned by Jdlrobson:

[mediawiki/skins/Vector@master] Test case for T310750

Reason:

https://gerrit.wikimedia.org/r/805911

Change 805911 restored by Clare Ming:

[mediawiki/skins/Vector@master] Test case for T310750

https://gerrit.wikimedia.org/r/805911

cjming moved this task from Doing to Code Review on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.
cjming subscribed.

Change 805911 merged by jenkins-bot:

[mediawiki/skins/Vector@master] Fix sticky header edit icons during A/B test init

https://gerrit.wikimedia.org/r/805911

Jdlrobson added a subscriber: Edtadros.

We'll enable this on the beta cluster so that we can test the beta cluster bucketing.

Once that's done users in the treatment bucket should see an edit icon in the sticky header.
Those in the control bucket should NOT see an edit icon in the sticky header.

Change 808995 had a related patch set uploaded (by Clare Ming; author: Clare Ming):

[operations/mediawiki-config@master] Enable sticky header edit test on beta cluster

https://gerrit.wikimedia.org/r/808995

cjming moved this task from QA to Code Review on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

Change 808995 merged by jenkins-bot:

[operations/mediawiki-config@master] Enable sticky header edit test on beta cluster

https://gerrit.wikimedia.org/r/808995

Mentioned in SAL (#wikimedia-operations) [2022-06-27T20:15:36Z] <cjming@deploy1002> Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:808995|Enable sticky header edit test on beta cluster (T310750)]] (duration: 03m 33s)

cjming moved this task from Code Review to QA on the Web-Team-Backlog (Kanbanana-FY-2021-22) board.

hi @Edtadros sticky header edit A/B test is deployed on beta cluster - you should be able to test there.
Per Jon's comment T310750#8030394 -- treatment groups sees edit icon, control does not

Test Result - Beta

Status: ✅ PASS
Environment: beta
OS: macOS Monterey
Browser: Chrome
Device: MBP
Emulated Device:NA

Test Artifact(s):

QA Steps

✅ AC1: Once that's done users in the treatment bucket should see an edit icon in the sticky header.

Screen Shot 2022-07-05 at 8.27.10 PM.png (1×1 px, 504 KB)

✅ AC2: Those in the control bucket should NOT see an edit icon in the sticky header.

Screen Shot 2022-07-05 at 8.25.30 PM.png (1×1 px, 473 KB)

Test Result - Prod (pilot wiki)

Status: ✅ PASS
Environment: frwiki
OS: macOS Monterey
Browser: Chrome
Device: MBP
Emulated Device:NA

Test Artifact(s):

QA Steps

✅ AC1: Once that's done users in the treatment bucket should see an edit icon in the sticky header.

Screen Shot 2022-07-11 at 6.53.58 AM.png (870×1 px, 239 KB)

✅ AC2: Those in the control bucket should NOT see an edit icon in the sticky header.

Screen Shot 2022-07-11 at 6.55.39 AM.png (867×1 px, 258 KB)