Details
Event Timeline
Change 854097 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis
Change 854097 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis
Mentioned in SAL (#wikimedia-operations) [2022-11-07T21:17:34Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:854097|Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353)]]
Mentioned in SAL (#wikimedia-operations) [2022-11-07T21:17:54Z] <urbanecm@deploy1002> urbanecm and matmarex: Backport for [[gerrit:854097|Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2022-11-07T21:23:22Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:854097|Enable wgDiscussionToolsEnablePermalinksBackend on group0 wikis (T315353)]] (duration: 05m 47s)
Change 854606 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis
Change 854606 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis
Mentioned in SAL (#wikimedia-operations) [2022-11-08T21:35:15Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:854606|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353)]]
Mentioned in SAL (#wikimedia-operations) [2022-11-08T21:35:34Z] <urbanecm@deploy1002> urbanecm and matmarex: Backport for [[gerrit:854606|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353)]] synced to the testservers: mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug2001.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2022-11-08T21:41:53Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:854606|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (T315353)]] (duration: 06m 36s)
Change 856566 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2)
Change 856566 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2)
Mentioned in SAL (#wikimedia-operations) [2022-11-14T14:23:05Z] <taavi@deploy1002> Started scap: Backport for [[gerrit:856566|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)]]
Mentioned in SAL (#wikimedia-operations) [2022-11-14T14:23:26Z] <taavi@deploy1002> taavi and matmarex: Backport for [[gerrit:856566|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2001.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2022-11-14T14:30:11Z] <taavi@deploy1002> Finished scap: Backport for [[gerrit:856566|Enable wgDiscussionToolsEnablePermalinksBackend on group1 wikis (#2) (T315353)]] (duration: 07m 05s)
I think I might have found the underlying problem here.
As I suspected, I think the DT updates (and probably any other secondary data update) are running twice in the same context.
For example, these two look similar if not almost identical:
- https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-deploy-1-7.0.0-1-2022.11.25?id=NaJUr4QBRsHAyp5Jv3PA
- https://logstash.wikimedia.org/app/discover#/doc/logstash-*/logstash-deploy-1-7.0.0-1-2022.11.25?id=8Exfr4QB-vZ6BagGXdDV
But look at the start of stack trace. One is explicit call to $page->doSecondaryDataUpdates( $options ); in the job and one is deferred update being ran implicitly after the job is done which JobExecuter runs regardless of whether secondary data updates are done or not (see JobExecutor.php(105)). The only other possible explanation is that we have another job that completely unrelatedly queues secondary data updates in its context.
That was a red herring but looking at the data, refreshlinks jobs are mostly useless and won't make any change to the output (assuming dt storage is filled) but ThreadItemStore::insertThreadItems is being called regardless, maybe we can do a short-circuit when revision id already exists in dt.
Change 901697 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis
Change 901697 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis
Mentioned in SAL (#wikimedia-operations) [2023-03-21T20:02:55Z] <taavi@deploy2002> Started scap: Backport for [[gerrit:900331|Enable DiscussionTools_visualenhancements_newsectionlink_enable on labs for testing]], [[gerrit:901697|Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis (T315353)]]
Mentioned in SAL (#wikimedia-operations) [2023-03-21T20:04:27Z] <taavi@deploy2002> esanders and taavi and matmarex: Backport for [[gerrit:900331|Enable DiscussionTools_visualenhancements_newsectionlink_enable on labs for testing]], [[gerrit:901697|Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis (T315353)]] synced to the testservers: mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug2002.codfw.wmnet
Mentioned in SAL (#wikimedia-operations) [2023-03-21T20:20:35Z] <taavi@deploy2002> Finished scap: Backport for [[gerrit:900331|Enable DiscussionTools_visualenhancements_newsectionlink_enable on labs for testing]], [[gerrit:901697|Enable wgDiscussionToolsEnablePermalinksBackend on group2 wikis (T315353)]] (duration: 17m 40s)
This is done now: all new comments being posted are being indexed in the permalink database. Old comments will be handled in T315510 by the maintenance script runs.
This ticket being resolved depends on permalinks being enabled and permalinks being enabled depends on T315510 being resolved.
Change 946998 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on s2/s3/s5/s6 group2
Change 946998 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on s2/s3/s5/s6 group2
Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:13:26Z] <urbanecm@deploy1002> Started scap: Backport for [[gerrit:946998|Enable wgDiscussionToolsEnablePermalinksBackend on s2/s3/s5/s6 group2 (T315353)]]
Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:14:57Z] <urbanecm@deploy1002> urbanecm and matmarex: Backport for [[gerrit:946998|Enable wgDiscussionToolsEnablePermalinksBackend on s2/s3/s5/s6 group2 (T315353)]] synced to the testservers mwdebug2002.codfw.wmnet, mwdebug1001.eqiad.wmnet, mwdebug1002.eqiad.wmnet, mwdebug2001.codfw.wmnet, and mw-debug kubernetes deployment (accessible via k8s-experimental XWD option)
Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:24:22Z] <urbanecm@deploy1002> Finished scap: Backport for [[gerrit:946998|Enable wgDiscussionToolsEnablePermalinksBackend on s2/s3/s5/s6 group2 (T315353)]] (duration: 10m 55s)
Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:29:13Z] <urbanecm> mwmaint1002: foreachwikiindblist 'group2 & s2' extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all --touched-after=20230615000000 (T315353)
Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:30:26Z] <urbanecm> mwmaint1002: foreachwikiindblist 'group2 & s3' extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all --touched-after=20230615000000 (T315353)
Mentioned in SAL (#wikimedia-operations) [2023-08-08T20:30:37Z] <urbanecm> mwmaint1002: foreachwikiindblist 'group2 & s5' extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --current --all --touched-after=20230615000000 (T315353)
Change 965562 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on s7 group2
Change 965562 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on s7 group2
Mentioned in SAL (#wikimedia-operations) [2023-10-12T20:58:22Z] <thcipriani@deploy2002> Started scap: Backport for [[gerrit:965562|Enable wgDiscussionToolsEnablePermalinksBackend on s7 group2 (T315353)]]
Mentioned in SAL (#wikimedia-operations) [2023-10-12T20:59:36Z] <thcipriani@deploy2002> thcipriani and matmarex: Backport for [[gerrit:965562|Enable wgDiscussionToolsEnablePermalinksBackend on s7 group2 (T315353)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
Mentioned in SAL (#wikimedia-operations) [2023-10-12T21:06:17Z] <thcipriani@deploy2002> Finished scap: Backport for [[gerrit:965562|Enable wgDiscussionToolsEnablePermalinksBackend on s7 group2 (T315353)]] (duration: 07m 55s)
Change #1036197 had a related patch set uploaded (by Bartosz Dziewoński; author: Bartosz Dziewoński):
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on enwiki (last one!)
Change #1036197 merged by jenkins-bot:
[operations/mediawiki-config@master] Enable wgDiscussionToolsEnablePermalinksBackend on enwiki
Mentioned in SAL (#wikimedia-operations) [2024-05-27T13:28:21Z] <logmsgbot> lucaswerkmeister-wmde@deploy1002 Started scap: Backport for [[gerrit:1036197|Enable wgDiscussionToolsEnablePermalinksBackend on enwiki (T315353)]], [[gerrit:1026511|Pre-emptively disable DiscussionToolsEnableThanks (no-op)]]
Mentioned in SAL (#wikimedia-operations) [2024-05-27T13:30:51Z] <logmsgbot> lucaswerkmeister-wmde@deploy1002 esanders and matmarex and lucaswerkmeister-wmde: Backport for [[gerrit:1036197|Enable wgDiscussionToolsEnablePermalinksBackend on enwiki (T315353)]], [[gerrit:1026511|Pre-emptively disable DiscussionToolsEnableThanks (no-op)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
Mentioned in SAL (#wikimedia-operations) [2024-05-27T13:46:37Z] <logmsgbot> lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for [[gerrit:1036197|Enable wgDiscussionToolsEnablePermalinksBackend on enwiki (T315353)]], [[gerrit:1026511|Pre-emptively disable DiscussionToolsEnableThanks (no-op)]] (duration: 18m 15s)