[go: up one dir, main page]

Page MenuHomePhabricator

Find flaky Selenium tests
Open, Needs TriagePublic

Description

For start, create patches for each repository running the tests suite 10 times.

RepositoryGerritJenkins Job (failures)quibble-vendor-mysql-php74-selenium-docker (failures)wmf-quibble-selenium-php74-docker (failures)MediaWiki-Docker (T256239)
mediawiki/core721790mediawiki-quibble-selenium-vendor-mysql-php72-docker P30977 0/100P30976 2/100 🚨P30863 0/100
mediawiki/extensions/AbuseFilter810352P30979 0/100P30980 0/100P30922 0/100
mediawiki/extensions/AdvancedSearch810354P30981 1/34 🚨
mediawiki/extensions/CampaignEvents836815P35361 2/100 🚨
mediawiki/extensions/CentralNotice
mediawiki/extensions/CirrusSearch
mediawiki/extensions/Cite810357P30982 0/100P30984 0/100P30924 0/100
mediawiki/extensions/CodeMirror810358P30985 0/100
mediawiki/extensions/Echo810359P30986 0/100P30987 0/100P30928 1/100 🚨
mediawiki/extensions/ElectronPdfService810360P30988 0/100P30929 0/100
mediawiki/extensions/EntitySchema810364P30989 0/53P30942 0/100
mediawiki/extensions/examples1055170P66810 0/100P66809 0/100
mediawiki/extensions/FileImporter810366P30842 0/100P30843 0/100
mediawiki/extensions/GlobalWatchlist810370P30991 1/100 🚨P30943 0/100
mediawiki/extensions/GrowthExperiments810371P30992 37/53 🚨P30993 36/52 🚨
mediawiki/extensions/Math810372P31827 0/100P31828 0/100P30944 0/100
mediawiki/extensions/MobileFrontend810373P31829 0/100P31830 0/100P30945 0/100
mediawiki/extensions/Newsletter810374P31836 0/100P30949 0/100
mediawiki/extensions/PageTriage993696P55812 6/100 🚨P55813 8/89 🚨
mediawiki/extensions/Popups810375P30964 0/100P30954 17/100 🚨 P30966 9/100 🚨
mediawiki/extensions/ProofreadPage810377P31838 0/100P31839 0/100P30967 0/100
mediawiki/extensions/RelatedArticles810378P31840 0/100P30968 0/100
mediawiki/extensions/RevisionSlider810379P31842 0/32P30969 0/100
mediawiki/extensions/TemplateWizard810820P31843 2/100 🚨P30972 8/100 🚨
mediawiki/extensions/TwoColConflict810821P31844 0/17P31000 1/100 🚨
mediawiki/extensions/VisualEditor807126P30857 0/100P30858 0/100P31410 0/100
mediawiki/extensions/Wikibase810831P31847 0/27P31849 0/24P31500 1/100 🚨
mediawiki/extensions/WikibaseLexeme810823P31865 0/11
mediawiki/extensions/WikiLambda810824P31866 0/100P31428 0/100
mediawiki/extensions/Wikistories810827P31867 0/100P31889 0/100
mediawiki/skins/MinervaNeue810828quibble-vendor-mysql-php72-docker P31868 0/100P31869 0/100P31896 1/100 🚨
wikibase/termbox810835
wikidata/query/gui810833generic-node14-browser-webdriver-docker P31715 0/100
wmde/wdio-wikibase

Notes

Details

SubjectRepoBranchLines +/-
mediawiki/coremaster+2 -1
mediawiki/extensions/examplesmaster+3 -2
mediawiki/extensions/CampaignEventsmaster+3 -1
mediawiki/extensions/PageTriagemaster+5 -5
mediawiki/extensions/PageTriagemaster+2 -1
mediawiki/extensions/VisualEditormaster+3 -2
mediawiki/coremaster+1 -1
mediawiki/skins/MinervaNeuemaster+3 -2
mediawiki/extensions/WikibaseLexememaster+3 -1
mediawiki/extensions/RelatedArticlesmaster+3 -2
mediawiki/extensions/MobileFrontendmaster+4 -2
mediawiki/extensions/CampaignEventsmaster+3 -2
mediawiki/extensions/FileImportermaster+3 -2
wikibase/termboxmaster+21 -0
mediawiki/extensions/Popupsmaster+3 -2
mediawiki/extensions/Citemaster+3 -2
mediawiki/extensions/AbuseFiltermaster+3 -2
mediawiki/extensions/CodeMirrormaster+3 -2
mediawiki/extensions/AdvancedSearchmaster+3 -1
mediawiki/extensions/ElectronPdfServicemaster+3 -2
mediawiki/extensions/GlobalWatchlistmaster+3 -2
mediawiki/extensions/EntitySchemamaster+3 -2
wikidata/query/guimaster+4 -1
wikimedia/portals/deploymaster+4 -1
mediawiki/extensions/Echomaster+3 -2
mediawiki/extensions/GrowthExperimentsmaster+2 -2
mediawiki/extensions/Wikistoriesmaster+3 -2
mediawiki/extensions/Mathmaster+3 -2
mediawiki/extensions/WikiLambdamaster+5 -2
mediawiki/extensions/Newslettermaster+3 -2
mediawiki/extensions/ProofreadPagemaster+3 -2
mediawiki/extensions/TemplateWizardmaster+3 -2
mediawiki/extensions/TwoColConflictmaster+3 -2
mediawiki/extensions/RevisionSlidermaster+3 -2
mediawiki/extensions/Wikibasemaster+4 -2
Show related patches Customize query in gerrit

Related Objects

StatusSubtypeAssignedTask
OpenNone
OpenNone
OpenNone
OpenNone
OpenNone
Resolvedkostajh
Resolvedovasileva
OpenNone
ResolvedOsamaahmed17
DuplicateNone
Resolvedmatmarex
OpenNone
OpenNone
OpenBUG REPORTNone
Resolvedkostajh
ResolvedBUG REPORTNone
OpenNone
OpenNone
OpenNone

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

https://integration.wikimedia.org/ci/job/wmf-quibble-selenium-php81-docker/1313/console#console-section-13 has some GrowthExperiment extension selenium tests time-out finding specific items (e.g.):

[0-0] Error in "add image.desktop: user can view image info and image details"
Error: element (".oo-ui-messageDialog-message") still not displayed after 5000ms

And also in general:

Error: Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)

After running the change through gate-and-submit again the same failures did not occur. Plus the selenium tests using other php versions passed without issues in the same attempt to get it through gate-and-submit.

Relatedly,

10:37:32 [Chrome 90.0.4430.212 linux #0-0] 1) add image mobile: user can close the image suggestion UI
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Error: Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at createTimeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/errors.js:498:15)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Test.Runnable._timeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:431:10)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Timeout.<anonymous> (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:246:24)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at listOnTimeout (internal/timers.js:557:17)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at processTimers (internal/timers.js:500:7)

Relatedly,

10:37:32 [Chrome 90.0.4430.212 linux #0-0] 1) add image mobile: user can close the image suggestion UI
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Error: Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at createTimeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/errors.js:498:15)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Test.Runnable._timeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:431:10)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Timeout.<anonymous> (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:246:24)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at listOnTimeout (internal/timers.js:557:17)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at processTimers (internal/timers.js:500:7)

Can you link to the build for this, please?

zeljkofilipin raised the priority of this task from Medium to Needs Triage.Feb 22 2023, 5:52 PM
zeljkofilipin moved this task from Q2 👔 to Backlog 🪒 on the User-zeljkofilipin board.

Change 866443 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Fix Special:Watchlist

Reason:

https://gerrit.wikimedia.org/r/866443

Change 721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

Reason:

https://gerrit.wikimedia.org/r/721790

Relatedly,

10:37:32 [Chrome 90.0.4430.212 linux #0-0] 1) add image mobile: user can close the image suggestion UI
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0] Error: Timeout of 60000ms exceeded. The execution in the test "add image mobile: user can close the image suggestion UI" took too long. Try to reduce the run time or increase your timeout for test specs (https://webdriver.io/docs/timeouts). (/workspace/src/extensions/GrowthExperiments/tests/selenium/specs/addimage.js)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at createTimeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/errors.js:498:15)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Test.Runnable._timeoutError (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:431:10)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at Timeout.<anonymous> (/workspace/src/extensions/GrowthExperiments/node_modules/mocha/lib/runnable.js:246:24)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at listOnTimeout (internal/timers.js:557:17)
10:37:32 [Chrome 90.0.4430.212 linux #0-0]     at processTimers (internal/timers.js:500:7)

Can you link to the build for this, please?

Created T334626 for this test

Change 807126 abandoned by Zfilipin:

[mediawiki/extensions/VisualEditor@master] WIP selenium: Run tests 100 times

Reason:

https://gerrit.wikimedia.org/r/807126

zeljkofilipin changed the task status from Open to In Progress.Jan 29 2024, 2:21 PM
zeljkofilipin claimed this task.
zeljkofilipin triaged this task as Low priority.
zeljkofilipin moved this task from Backlog 🪒 to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin updated the task description. (Show Details)

Change 993696 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/PageTriage@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/993696

zeljkofilipin updated the task description. (Show Details)
zeljkofilipin updated the task description. (Show Details)

Change 994181 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/PageTriage@master] WIP selenium: Fix flaky test

https://gerrit.wikimedia.org/r/994181

Are you finding that missing awaits are a common cause of flaky Selenium tests? Would it make sense to set up some CI static code analysis that fails Selenium tests that don't await certain pre-programmed methods such as .click() and .isExisting()?

Change 993696 abandoned by Zfilipin:

[mediawiki/extensions/PageTriage@master] WIP selenium: Find flaky tests

Reason:

just a test

https://gerrit.wikimedia.org/r/993696

Change 994181 merged by jenkins-bot:

[mediawiki/extensions/PageTriage@master] selenium: Fix flaky test

https://gerrit.wikimedia.org/r/994181

zeljkofilipin changed the task status from In Progress to Open.Feb 9 2024, 1:24 PM
zeljkofilipin removed zeljkofilipin as the assignee of this task.
zeljkofilipin raised the priority of this task from Low to Needs Triage.
zeljkofilipin moved this task from Deep work 🌊 to Backlog 🪒 on the User-zeljkofilipin board.

Change 721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

zeljkofilipin changed the task status from Open to In Progress.Feb 19 2024, 12:47 PM
zeljkofilipin claimed this task.
zeljkofilipin triaged this task as Low priority.
zeljkofilipin moved this task from Backlog 🪒 to Deep work 🌊 on the User-zeljkofilipin board.
zeljkofilipin changed the task status from In Progress to Open.Feb 19 2024, 5:29 PM
zeljkofilipin removed zeljkofilipin as the assignee of this task.
zeljkofilipin raised the priority of this task from Low to Needs Triage.
zeljkofilipin moved this task from Deep work 🌊 to Backlog 🪒 on the User-zeljkofilipin board.

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

Reason:

https://gerrit.wikimedia.org/r/721790

Change #1019084 had a related patch set uploaded (by Vaughn Walters; author: Vaughn Walters):

[mediawiki/extensions/CampaignEvents@master] selenium: Skipping test that fails on betacluster

https://gerrit.wikimedia.org/r/1019084

Change #1019084 merged by jenkins-bot:

[mediawiki/extensions/CampaignEvents@master] selenium: Skipping test that fails on betacluster

https://gerrit.wikimedia.org/r/1019084

Change #721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #1055170 had a related patch set uploaded (by Zfilipin; author: Zfilipin):

[mediawiki/extensions/examples@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/1055170

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #1055170 abandoned by Zfilipin:

[mediawiki/extensions/examples@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/1055170

Change #721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 restored by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790

Change #721790 abandoned by Zfilipin:

[mediawiki/core@master] WIP selenium: Find flaky tests

https://gerrit.wikimedia.org/r/721790