[go: up one dir, main page]
Page MenuHomePhabricator
Create Task
Maniphest T231961

DBPerformance warning: "Expectation masterConns <= 0 not met" from CentralAuth special pages
Open, Needs TriagePublic
Actions

Description

The DBPerformance contains several warnings about unexpected DB_MASTER connections on GET requests from CentralAuth special pages.

Logstash: https://logstash.wikimedia.org/goto/80e6ec068512bab1687d248131bbee51

  • CentralAuthGroupMembershipProxy
  • Special:GlobalRenameRequest
  • Special:CentralLogin
  • Special:GlobalRenameQueue
CentralAuthGroupMembershipProxy
  • Request ID: XW8QQwpAADkAAGvZccAAAAAF
  • Request URL: GET /w/index.php?user=Krinkle&title=Special%3AGlobalUserRights
Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1): [connect to 10.64.48.15 (centralauth)]
#5 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(269): CentralAuthUtils::getCentralDB()
#6 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(472): CentralAuthUser->getSafeReadDB()
#7 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(415): CentralAuthUser->loadFromDatabase()
#8 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(588): CentralAuthUser->loadState()
#9 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(685): CentralAuthUser->getId()
#10 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthGroupMembershipProxy.php(50): CentralAuthUser->exists()
#11 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalGroupMembership.php(112): CentralAuthGroupMembershipProxy::newFromName()
#12 /srv/mediawiki/php-1.34.0-wmf.20/includes/specials/SpecialUserrights.php(108): SpecialGlobalGroupMembership->fetchUser()
#13 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/SpecialPage.php(573): UserrightsPage->execute() …
#18 /srv/mediawiki/php-1.34.0-wmf.20/index.php(42): MediaWiki->run()

The caller doesn't appear to need a master DB (not sure?) and isn't specifically asking for it. But, it uses the CentralAuthGroupMembershipProxy class which is hardcoded to obtain a master connection. That seems problematic.

Special:GlobalRenameRequest
  • Request ID: XW8ZGQpAICAAACgl7RwAAABE
  • Request URL: GET /wiki/Special:GlobalRenameRequest
Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1): [connect to 10.64.48.15 (centralauth)]
#5 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(269): CentralAuthUtils::getCentralDB()
#6 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(472): CentralAuthUser->getSafeReadDB()
#7 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(415): CentralAuthUser->loadFromDatabase()
#8 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(588): CentralAuthUser->loadState()
#9 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(685): CentralAuthUser->getId()
#10 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/GlobalRename/GlobalRenameRequest.php(434): CentralAuthUser->exists()
#11 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameRequest.php(211): GlobalRenameRequest::isNameAvailable()
#12 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameRequest.php(152): SpecialGlobalRenameRequest->suggestedUsername()
#13 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/FormSpecialPage.php(121): SpecialGlobalRenameRequest->getFormFields()
#14 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/FormSpecialPage.php(183): FormSpecialPage->getForm()
#15 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameRequest.php(89): FormSpecialPage->execute()
#16 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/SpecialPage.php(573): SpecialGlobalRenameRequest->execute() …
#21 /srv/mediawiki/php-1.34.0-wmf.20/index.php(42): MediaWiki->run()

Upon opening the special page form (viewing, not submitting), it calls GlobalRenameRequest::isNameAvailable which is hardwired to obtain a master instance. If don't know if it relies on that, but either that method or the call to it needs to change.

Special:CentralLogin
  • Request ID: XW8VAgpAADkAAFJxY88AAABR
  • Request URL: loginwiki GET /wiki/Special:CentralLogin/start?token=…&cpPosIndex=1 (referer ja.wikipedia.org)
Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 2): [connect to 10.64.48.15 (centralauth)] …
#5 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(269): CentralAuthUtils::getCentralDB()
#6 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(472): CentralAuthUser->getSafeReadDB()
#7 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(415): CentralAuthUser->loadFromDatabase()
#8 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(588): CentralAuthUser->loadState()
#9 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(685): CentralAuthUser->getId()
#10 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialCentralLogin.php(79): CentralAuthUser->exists()
#11 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialCentralLogin.php(97): Closure$SpecialCentralLogin::doLoginStart()
#12 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialCentralLogin.php(50): SpecialCentralLogin->doLoginStart()
#13 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/SpecialPage.php(573): SpecialCentralLogin->execute() …
#18 /srv/mediawiki/php-1.34.0-wmf.20/index.php(42): MediaWiki->run()

In SpecialCentralLogin->doLoginStart, a check is made for the replica DB, and if the user is not found, it then retries from a master DB. This seems intentional, but I'm not sure it is still needed.

For one, chronology protector should be taking care of this already by ensuring we have a replica that has caught up with the events of the user sign up. But the, why is it reaching the else clause? Perhaps this is an error case where the user doesn't exist at all. If so, then it'd be preferable not to handle errors by burdening the master with an impossible query. Alternatively, if we do need this and it's working fine, it should probably opt-out from the TrxProfiler the same way that e.g. RollbackAction does.

Special:GlobalRenameQueue
  • Request ID: XW63AgpAAEUAAFT6tf8AAACL
  • Request URL: GET /wiki/Special:GlobalRenameQueue/request/ …<number>… 
Expectation (masterConns <= 0) by MediaWiki::main not met (actual: 1): [connect to 10.64.48.15 (centralauth)] …
#5 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(269): CentralAuthUtils::getCentralDB()
#6 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(472): CentralAuthUser->getSafeReadDB()
#7 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(415): CentralAuthUser->loadFromDatabase()
#8 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/CentralAuthUser.php(705): CentralAuthUser->loadState()
#9 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameUser.php(227): CentralAuthUser->isHidden()
#10 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameQueue.php(427): SpecialGlobalRenameUser->processAntiSpoofConflicts()
#11 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameQueue.php(244): SpecialGlobalRenameQueue->doShowProcessForm()
#12 /srv/mediawiki/php-1.34.0-wmf.20/extensions/CentralAuth/includes/specials/SpecialGlobalRenameQueue.php(74): SpecialGlobalRenameQueue->handleProcessRequest()
#13 /srv/mediawiki/php-1.34.0-wmf.20/includes/specialpage/SpecialPage.php(573): SpecialGlobalRenameQueue->execute() …
#19 /srv/mediawiki/w/index.php(3): include()

The SpecialGlobalRenameUser->processAntiSpoofConflicts method is hardwired to obtain a master connection. Either needs to change, support both, or not be used when displaying the page in question.

See also:

Details

SubjectRepoBranchLines +/-
GlobalRenameRequest: use replica db to validate suggestionsmediawiki/extensions/CentralAuthmaster+13 -4
AntiSpoof: use replica db for displaying conflictsmediawiki/extensions/CentralAuthmaster+1 -1
Customize query in gerrit

Related Objects
Search...

Event Timeline

Krinkle created this task.Sep 4 2019, 2:25 AM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptSep 4 2019, 2:25 AM
Krinkle moved this task from Limbo to Perf recommendation on the Performance-Team (Radar) board.Sep 4 2019, 2:25 AM
Krinkle added a project: Sustainability (MediaWiki-MultiDC).
Krinkle updated the task description. (Show Details)Sep 4 2019, 2:29 AM
Krinkle updated the task description. (Show Details)
Krinkle mentioned this in T252244: CentralAuth extension: Code stewardship review.May 8 2020, 8:20 PM
taavi subscribed.Dec 28 2020, 12:43 PM

Notes about some research I did into the CentralAuthGroupMembershipProxy/SpecialGlobalGroupMembership issue:

  • It's based on UserrightsPage/SpecialUserrights.php
  • ::fetchUser( string $username, bool $writing = true ) is used to create membership proxy objects, $writing is always true
    • The function is called even when you have not specified a username, it attempts to guard against ($username === '' but in those cases $username is null)
    • $writing isn't used on this implementation at all, and on SpecialUserrights the only use is to validate userrights-interwiki user right
    • UserrightsProxy/UserrightsPage also always creates DB master connections when doing anything with interwiki userrights
    • side note: why does ApiQueryUsers call this method? this should probably extracted into its own service?
  • I don't think it doesn't matter if we load the user rights from a replica when viewing the page, master connection is required when updating user rights
taavi mentioned this in T283365: CentralAuthUser is sometimes loaded from the primary database on GET requests.May 21 2021, 3:27 PM
Tchanders added a parent task: T267945: Increase in DBPerformance warnings since 1.36.0-wmf.13.May 21 2021, 3:34 PM
Tchanders merged a task: T283365: CentralAuthUser is sometimes loaded from the primary database on GET requests.
Tchanders subscribed.
Tchanders added a comment.May 21 2021, 3:39 PM

From the duplicate task:

There are a number of different triggers, but the stack traces all involve calls to CentralAuthUser::getSafeReadDB, which chooses primary or replica depending on whether changes have been made in the primary database, regardless of the HTTP request method:

/**
 * @return IDatabase Master or replica based on shouldUseMasterDB()
 * @throws CentralAuthReadOnlyError
 */
protected function getSafeReadDB() {
	return $this->shouldUseMasterDB()
		? CentralAuthUtils::getCentralDB()
		: CentralAuthUtils::getCentralReplicaDB();
}

/**
 * Get (and init if needed) the value of mFromMaster
 *
 * @return bool
 */
protected function shouldUseMasterDB() {
	if ( $this->mFromMaster === null ) {
		// Calls LoadBalancer::hasOrMadeRecentMasterChanges
		$this->mFromMaster = self::centralLBHasRecentMasterChanges();
	}

	return $this->mFromMaster;
}

Another example that triggers this warning involves the hook handler CentralAuthHooks::onGetUserBlock:

from /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/TransactionProfiler.php(444)
#0 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/TransactionProfiler.php(224): Wikimedia\Rdbms\TransactionProfiler->reportExpectationViolated(string, string, integer)
#1 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1010): Wikimedia\Rdbms\TransactionProfiler->recordConnection(string, string, boolean)
#2 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(964): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#3 /srv/mediawiki/php-1.37.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1103): Wikimedia\Rdbms\LoadBalancer->getConnection(integer, array, string, integer)
#4 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUtilityService.php(110): Wikimedia\Rdbms\LoadBalancer->getConnectionRef(integer, array, string)
#5 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUtils.php(36): CentralAuthUtilityService->getCentralDB()
#6 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(273): CentralAuthUtils::getCentralDB()
#7 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(2378): CentralAuthUser->getSafeReadDB()
#8 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(594): CentralAuthUser->loadAttached()
#9 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(573): CentralAuthUser->loadFromCacheObject(array)
#10 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(417): CentralAuthUser->loadFromCache()
#11 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(605): CentralAuthUser->loadState()
#12 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthUser.php(702): CentralAuthUser->getId()
#13 /srv/mediawiki/php-1.37.0-wmf.6/extensions/CentralAuth/includes/CentralAuthHooks.php(958): CentralAuthUser->exists()
#14 /srv/mediawiki/php-1.37.0-wmf.6/includes/HookContainer/HookContainer.php(330): CentralAuthHooks::onGetUserBlock(User, string, NULL)
#15 /srv/mediawiki/php-1.37.0-wmf.6/includes/HookContainer/HookContainer.php(137): MediaWiki\HookContainer\HookContainer->callLegacyHook(string, array, array, array)
#16 /srv/mediawiki/php-1.37.0-wmf.6/includes/HookContainer/HookRunner.php(1961): MediaWiki\HookContainer\HookContainer->run(string, array)
#17 /srv/mediawiki/php-1.37.0-wmf.6/includes/block/BlockManager.php(175): MediaWiki\HookContainer\HookRunner->onGetUserBlock(User, string, NULL)
#18 /srv/mediawiki/php-1.37.0-wmf.6/includes/user/User.php(1611): MediaWiki\Block\BlockManager->getUserBlock(User, WebRequest, boolean, boolean)
#19 /srv/mediawiki/php-1.37.0-wmf.6/includes/user/User.php(1906): User->getBlockedStatus(boolean, boolean)
#20 /srv/mediawiki/php-1.37.0-wmf.6/includes/block/BlockManager.php(484): User->getBlock()
#21 /srv/mediawiki/php-1.37.0-wmf.6/includes/MediaWiki.php(780): MediaWiki\Block\BlockManager->trackBlockWithCookie(User, WebResponse)
#22 /srv/mediawiki/php-1.37.0-wmf.6/includes/MediaWiki.php(646): MediaWiki::preOutputCommit(RequestContext, Closure)
#23 /srv/mediawiki/php-1.37.0-wmf.6/includes/MediaWiki.php(940): MediaWiki->doPreOutputCommit(Closure)
#24 /srv/mediawiki/php-1.37.0-wmf.6/includes/MediaWiki.php(551): MediaWiki->main()
#25 /srv/mediawiki/php-1.37.0-wmf.6/index.php(53): MediaWiki->run()
#26 /srv/mediawiki/php-1.37.0-wmf.6/index.php(46): wfIndexMain()
#27 /srv/mediawiki/w/index.php(3): require(string)
#28 {main}
gerritbot added a comment.Jan 17 2022, 3:48 PM

Change 754534 had a related patch set uploaded (by Majavah; author: Majavah):

[mediawiki/extensions/CentralAuth@master] AntiSpoof: use replica db for displaying conflicts

https://gerrit.wikimedia.org/r/754534

gerritbot added a project: Patch-For-Review.Jan 17 2022, 3:48 PM
gerritbot added a comment.Jan 17 2022, 3:59 PM

Change 754537 had a related patch set uploaded (by Majavah; author: Majavah):

[mediawiki/extensions/CentralAuth@master] GlobalRenameRequest: use replica db to validate suggestions

https://gerrit.wikimedia.org/r/754537

gerritbot added a comment.Jan 17 2022, 5:22 PM

Change 754534 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] AntiSpoof: use replica db for displaying conflicts

https://gerrit.wikimedia.org/r/754534

ReleaseTaggerBot added a project: MW-1.38-notes (1.38.0-wmf.18; 2022-01-17).Jan 17 2022, 6:00 PM
gerritbot added a comment.Jan 18 2022, 7:30 PM

Change 754537 merged by jenkins-bot:

[mediawiki/extensions/CentralAuth@master] GlobalRenameRequest: use replica db to validate suggestions

https://gerrit.wikimedia.org/r/754537

ReleaseTaggerBot edited projects, added MW-1.38-notes (1.38.0-wmf.19; 2022-01-24); removed MW-1.38-notes (1.38.0-wmf.18; 2022-01-17).Jan 18 2022, 8:00 PM
Maintenance_bot removed a project: Patch-For-Review.Jan 18 2022, 8:10 PM
Krinkle edited projects, added Wikimedia-Performance-recommendation; removed Performance-Team (Radar).Aug 18 2023, 8:41 PM
Restricted Application added a project: MediaWiki-Platform-Team. · View Herald TranscriptAug 18 2023, 8:42 PM
larissagaulia moved this task from Inbox, needs triage to Backlog: non-prioritized on the MediaWiki-Platform-Team board.Aug 22 2023, 10:16 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits