[go: up one dir, main page]
Page MenuHomePhabricator
Create Task
Maniphest T373086

PHP Warning: Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value.
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Error
labels.normalized_message
[{reqId}] {exception_url}   PHP Warning: Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value.
error.stack_trace
from /srv/mediawiki/php-1.43.0-wmf.19/includes/libs/Stats/Metrics/CounterMetric.php(84)
#0 [internal function]: MWExceptionHandler::handleError(int, string, string, int, array)
#1 /srv/mediawiki/php-1.43.0-wmf.19/includes/libs/Stats/Metrics/CounterMetric.php(84): trigger_error(string, int)
#2 /srv/mediawiki/php-1.43.0-wmf.19/extensions/EventBus/includes/EventBus.php(221): Wikimedia\Stats\Metrics\CounterMetric->incrementBy(float)
#3 /srv/mediawiki/php-1.43.0-wmf.19/extensions/EventBus/includes/EventBus.php(507): MediaWiki\Extension\EventBus\EventBus->incrementMetricByValue(string, int, array, array)
#4 /srv/mediawiki/php-1.43.0-wmf.19/extensions/EventBus/includes/Adapters/RCFeed/EventBusRCFeedEngine.php(57): MediaWiki\Extension\EventBus\EventBus->send(array)
#5 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/MWCallableUpdate.php(52): MediaWiki\Extension\EventBus\Adapters\RCFeed\EventBusRCFeedEngine::MediaWiki\Extension\EventBus\Adapters\RCFeed\{closure}(string)
#6 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdates.php(495): MediaWiki\Deferred\MWCallableUpdate->doUpdate()
#7 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdates.php(207): MediaWiki\Deferred\DeferredUpdates::attemptUpdate(MediaWiki\Deferred\MWCallableUpdate)
#8 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdates.php(305): MediaWiki\Deferred\DeferredUpdates::run(MediaWiki\Deferred\MWCallableUpdate)
#9 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdatesScope.php(269): MediaWiki\Deferred\DeferredUpdates::MediaWiki\Deferred\{closure}(MediaWiki\Deferred\MWCallableUpdate, int)
#10 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdatesScope.php(198): MediaWiki\Deferred\DeferredUpdatesScope->processStageQueue(int, int, Closure)
#11 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdates.php(308): MediaWiki\Deferred\DeferredUpdatesScope->processUpdates(int, Closure)
#12 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdatesScope.php(269): MediaWiki\Deferred\DeferredUpdates::MediaWiki\Deferred\{closure}(MediaWiki\Deferred\MWCallableUpdate, int)
#13 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdatesScope.php(198): MediaWiki\Deferred\DeferredUpdatesScope->processStageQueue(int, int, Closure)
#14 /srv/mediawiki/php-1.43.0-wmf.19/includes/deferred/DeferredUpdates.php(313): MediaWiki\Deferred\DeferredUpdatesScope->processUpdates(int, Closure)
#15 /srv/mediawiki/php-1.43.0-wmf.19/includes/MediaWikiEntryPoint.php(674): MediaWiki\Deferred\DeferredUpdates::doUpdates()
#16 /srv/mediawiki/php-1.43.0-wmf.19/includes/MediaWikiEntryPoint.php(496): MediaWiki\MediaWikiEntryPoint->restInPeace()
#17 /srv/mediawiki/php-1.43.0-wmf.19/includes/MediaWikiEntryPoint.php(454): MediaWiki\MediaWikiEntryPoint->doPostOutputShutdown()
#18 /srv/mediawiki/php-1.43.0-wmf.19/includes/MediaWikiEntryPoint.php(209): MediaWiki\MediaWikiEntryPoint->postOutputShutdown()
#19 /srv/mediawiki/php-1.43.0-wmf.19/index.php(58): MediaWiki\MediaWikiEntryPoint->run()
#20 /srv/mediawiki/w/index.php(3): require(string)
#21 {main}
Impact

Low; 11 occurrences, first one on Aug 2, 2024 @ 21:27:01.017 (with bogus values), first one with non-bogus values on hewiki being on 1.43.0-wmf.19 on Aug 21, 2024 @ 15:51:48.425.

Notes

Details

MediaWiki Version
1.43.0-wmf.16
Request URL
https://it.wikipedia.org/wiki/Speciale:Blocca/*?gbdisableUTEdit=*&gbexpiry=*&gbnamespaces=*&gbpages=*&gbpartial=*&gbreason=*
SubjectRepoBranchLines +/-
Checker: ensure all labels are setmediawiki/extensions/CirrusSearchwmf/1.43.0-wmf.21+10 -6
Checker: ensure all labels are setmediawiki/extensions/CirrusSearchmaster+10 -6
metrics: provide a fallback value for emtpy labelsmediawiki/extensions/EventBusmaster+14 -6
Customize query in gerrit

Related Objects
Search...

Event Timeline

• Aklapper created this task.Aug 22 2024, 10:59 AM
colewhite edited projects, added Event-Platform; removed MediaWiki-libs-Stats.Tue, Aug 27, 11:02 PM
Restricted Application added a project: Data-Engineering. · View Herald TranscriptTue, Aug 27, 11:02 PM
colewhite added subscribers: gmodena, Ottomata, colewhite.EditedTue, Aug 27, 11:03 PM

Introduced in https://gerrit.wikimedia.org/r/c/mediawiki/extensions/EventBus/+/1049831

@gmodena @Ottomata

gmodena claimed this task.Wed, Aug 28, 8:49 AM
gmodena moved this task from Incoming (new tickets) to Q1 2024 July 1st - September 30th on the Data-Engineering board.Wed, Aug 28, 8:52 AM
gmodena edited projects, added Data-Engineering (Q1 2024 July 1st - September 30th); removed Data-Engineering.
gmodena moved this task from Next Up to In progress on the Data-Engineering (Q1 2024 July 1st - September 30th) board.
gmodena added a comment.Wed, Aug 28, 9:33 AM

@colewhite ack. Thanks for the heads up.

gerritbot added a comment.Wed, Aug 28, 1:26 PM

Change #1067997 had a related patch set uploaded (by Gmodena; author: Gmodena):

[mediawiki/extensions/EventBus@master] metrics: provide a fallback value for emtpy labels

https://gerrit.wikimedia.org/r/1067997

gerritbot added a project: Patch-For-Review.Wed, Aug 28, 1:26 PM
gerritbot added a comment.Wed, Aug 28, 2:32 PM

Change #1067997 merged by jenkins-bot:

[mediawiki/extensions/EventBus@master] metrics: provide a fallback value for emtpy labels

https://gerrit.wikimedia.org/r/1067997

gmodena moved this task from In progress to Done on the Data-Engineering (Q1 2024 July 1st - September 30th) board.Wed, Aug 28, 2:34 PM
ReleaseTaggerBot added a project: MW-1.43-notes (1.43.0-wmf.21; 2024-09-03).Wed, Aug 28, 3:00 PM
thcipriani moved this task from Untriaged to Aug 2024 on the Wikimedia-production-error board.Thu, Aug 29, 3:26 PM
dancy subscribed.Tue, Sep 3, 3:06 PM

https://gerrit.wikimedia.org/r/1067997 was included in this week's train branch (wmf/1.43.0-wmf.21) but I still see many Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value and Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value warnings being logged for testwikidatawiki or test.wikidata.org today.

colewhite added a project: CirrusSearch.Tue, Sep 3, 3:25 PM

Coming from CirrusSearch extension:

from /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/Stats/Metrics/CounterMetric.php(84)
#0 [internal function]: MWExceptionHandler::handleError(int, string, string, int, array)
#1 /srv/mediawiki/php-1.43.0-wmf.21/includes/libs/Stats/Metrics/CounterMetric.php(84): trigger_error(string, int)
#2 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Sanity/Checker.php(184): Wikimedia\Stats\Metrics\CounterMetric->incrementBy(float)
#3 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Api/CheckSanity.php(104): CirrusSearch\Sanity\Checker->check(array)
#4 /srv/mediawiki/php-1.43.0-wmf.21/extensions/CirrusSearch/includes/Api/CheckSanity.php(60): CirrusSearch\Api\CheckSanity->check(CirrusSearch\Sanity\Checker, int, int)
#5 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(1948): CirrusSearch\Api\CheckSanity->execute()
#6 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(926): ApiMain->executeAction()
#7 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiMain.php(897): ApiMain->executeActionWithErrorHandling()
#8 /srv/mediawiki/php-1.43.0-wmf.21/includes/api/ApiEntryPoint.php(153): ApiMain->execute()
#9 /srv/mediawiki/php-1.43.0-wmf.21/includes/MediaWikiEntryPoint.php(200): MediaWiki\Api\ApiEntryPoint->execute()
#10 /srv/mediawiki/php-1.43.0-wmf.21/api.php(44): MediaWiki\MediaWikiEntryPoint->run()
#11 /srv/mediawiki/w/api.php(3): require(string)
#12 {main}
Restricted Application added a project: Discovery-Search. · View Herald TranscriptTue, Sep 3, 3:25 PM
Maintenance_bot removed a project: Patch-For-Review.Tue, Sep 3, 3:30 PM
gerritbot added a comment.Tue, Sep 3, 4:22 PM

Change #1070298 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@master] Checker: ensure all labels are set

https://gerrit.wikimedia.org/r/1070298

gerritbot added a project: Patch-For-Review.Tue, Sep 3, 4:22 PM
dancy triaged this task as Unbreak Now! priority.Tue, Sep 3, 6:15 PM
dancy added a parent task: T373640: 1.43.0-wmf.21 deployment blockers.
dancy added a comment.EditedTue, Sep 3, 6:17 PM

The warning rate ramped up a lot (~860/15 minutes) after rolling 1.43.0-wmf.21 to group0 today, so I'm making this task a train blocker.

gerritbot added a comment.Tue, Sep 3, 8:05 PM

Change #1070298 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@master] Checker: ensure all labels are set

https://gerrit.wikimedia.org/r/1070298

Maintenance_bot removed a project: Patch-For-Review.Tue, Sep 3, 8:30 PM
ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.22; 2024-09-10); removed MW-1.43-notes (1.43.0-wmf.21; 2024-09-03).Tue, Sep 3, 9:00 PM
gerritbot added a comment.Wed, Sep 4, 7:46 AM

Change #1070531 had a related patch set uploaded (by DCausse; author: DCausse):

[mediawiki/extensions/CirrusSearch@wmf/1.43.0-wmf.21] Checker: ensure all labels are set

https://gerrit.wikimedia.org/r/1070531

gerritbot added a project: Patch-For-Review.Wed, Sep 4, 7:46 AM
gmodena added a subscriber: dcausse.Wed, Sep 4, 8:34 AM

@dancy @dcausse

Ack. Investigating.
My original patch only addressed EventBus. I can't seem to find any new warn entry for that service in logstash.

Looks like the spike is only related to CirrusSearch?

dcausse added a comment.Wed, Sep 4, 8:39 AM
In T373086#10116237, @gmodena wrote:

Looks like the spike is only related to CirrusSearch?

yes this is my understanding as well, it is quite noisy so it might hide others, I'm shipping a fix so hopefully we'll get some more clarity soon.

dcausse claimed this task.Wed, Sep 4, 8:39 AM
gerritbot added a comment.Wed, Sep 4, 8:51 AM

Change #1070531 merged by jenkins-bot:

[mediawiki/extensions/CirrusSearch@wmf/1.43.0-wmf.21] Checker: ensure all labels are set

https://gerrit.wikimedia.org/r/1070531

Stashbot added a comment.Wed, Sep 4, 8:52 AM

Mentioned in SAL (#wikimedia-operations) [2024-09-04T08:52:40Z] <dcausse@deploy1003> Started scap sync-world: Backport for [[gerrit:1070531|Checker: ensure all labels are set (T373086)]]

Stashbot added a comment.Wed, Sep 4, 8:54 AM

Mentioned in SAL (#wikimedia-operations) [2024-09-04T08:54:52Z] <dcausse@deploy1003> dcausse: Backport for [[gerrit:1070531|Checker: ensure all labels are set (T373086)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Stashbot added a comment.Wed, Sep 4, 8:59 AM

Mentioned in SAL (#wikimedia-operations) [2024-09-04T08:59:29Z] <dcausse@deploy1003> Finished scap sync-world: Backport for [[gerrit:1070531|Checker: ensure all labels are set (T373086)]] (duration: 06m 48s)

ReleaseTaggerBot edited projects, added MW-1.43-notes (1.43.0-wmf.21; 2024-09-03); removed MW-1.43-notes (1.43.0-wmf.22; 2024-09-10).Wed, Sep 4, 9:00 AM
dcausse closed this task as Resolved.Wed, Sep 4, 9:10 AM
dcausse reassigned this task from dcausse to gmodena.

Seems to be fixed, tentatively closing

Maintenance_bot removed a project: Patch-For-Review.Wed, Sep 4, 9:30 AM
gmodena added a comment.Wed, Sep 4, 9:41 AM
In T373086#10116254, @dcausse wrote:
In T373086#10116237, @gmodena wrote:

Looks like the spike is only related to CirrusSearch?

yes this is my understanding as well, it is quite noisy so it might hide others, I'm shipping a fix so hopefully we'll get some more clarity soon.

Just an ack that I don't see any EventBus (or other extensions) entry in logstash.

Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits