After moving these tasks to shellbox and pointing the jobqueue back to k8s jobrunners, these errors have not reappeared.

Change 989128 merged by jenkins-bot:

[operations/deployment-charts@master] jobqueue: restore media handling jobs to k8s

https://gerrit.wikimedia.org/r/989128

Change 989128 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] jobqueue: restore media handling jobs to k8s

https://gerrit.wikimedia.org/r/989128

Mentioned in SAL (#wikimedia-operations) [2024-01-09T09:54:02Z] <oblivian@deploy2002> Finished scap: Backport for [[gerrit:987033|Always process media files via shellbox on k8s (T352515)]] (duration: 11m 03s)

Mentioned in SAL (#wikimedia-operations) [2024-01-09T09:44:41Z] <oblivian@deploy2002> oblivian: Backport for [[gerrit:987033|Always process media files via shellbox on k8s (T352515)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-09T09:42:59Z] <oblivian@deploy2002> Started scap: Backport for [[gerrit:987033|Always process media files via shellbox on k8s (T352515)]]

Change 987033 merged by jenkins-bot:

[operations/mediawiki-config@master] Always process media files via shellbox on k8s

https://gerrit.wikimedia.org/r/987033

Mentioned in SAL (#wikimedia-operations) [2024-01-09T09:27:22Z] <oblivian@deploy2002> Finished scap: Backport for [[gerrit:987032|Use shellbox for djvu handling on kubernetes (T352515)]] (duration: 23m 56s)

Mentioned in SAL (#wikimedia-operations) [2024-01-09T09:05:17Z] <oblivian@deploy2002> oblivian: Backport for [[gerrit:987032|Use shellbox for djvu handling on kubernetes (T352515)]] synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)

Mentioned in SAL (#wikimedia-operations) [2024-01-09T09:03:25Z] <oblivian@deploy2002> Started scap: Backport for [[gerrit:987032|Use shellbox for djvu handling on kubernetes (T352515)]]

Change 987032 merged by jenkins-bot:

[operations/mediawiki-config@master] Use shellbox for djvu handling on kubernetes

https://gerrit.wikimedia.org/r/987032

Change 987432 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/mediawiki-config@master] Explicitly disable all local imagescaling on k8s

https://gerrit.wikimedia.org/r/987432

Change 987033 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/mediawiki-config@master] Always process media files via shellbox on k8s

https://gerrit.wikimedia.org/r/987033

Change 987032 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[operations/mediawiki-config@master] Use shellbox for djvu handling on kubernetes

https://gerrit.wikimedia.org/r/987032

In T352515#9421751, @thcipriani wrote:

In T352515#9415206, @Clement_Goubert wrote:

We've moved the affected job (AssembleUploadChunks) back to metal temporarily.

There are still some of these errors popping up for frwikisource Special:Téléverser (Special:Upload), but not on other wikis as far as I can tell.

Note that some of the PDFHandler and PagedTiffHandler errors are still showing up on the mw-jobrunner k8s hosts. They seem to be coming from the PublishStashedFileJob. Mostly on commonswiki.

For example:

Error

• mwversion: 1.42.0-wmf.10
• reqId: 0cb37d30-a37b-49e6-88f2-e14eb1290140
• Find reqId in Logstash

normalized_message

[{reqId}] {exception_url}   RuntimeException: firejail is enabled, but cannot be found

exception.trace

from /srv/mediawiki/php-1.42.0-wmf.10/includes/shell/CommandFactory.php(125)
#0 /srv/mediawiki/php-1.42.0-wmf.10/includes/shell/CommandFactory.php(189): MediaWiki\Shell\CommandFactory->getLocalShellboxOptions()
#1 /srv/mediawiki/php-1.42.0-wmf.10/extensions/PdfHandler/includes/PdfImage.php(110): MediaWiki\Shell\CommandFactory->createBoxed(string)
#2 /srv/mediawiki/php-1.42.0-wmf.10/extensions/PdfHandler/includes/PdfHandler.php(280): MediaWiki\Extension\PdfHandler\PdfImage->retrieveMetaData()
#3 /srv/mediawiki/php-1.42.0-wmf.10/includes/media/MediaHandler.php(227): MediaWiki\Extension\PdfHandler\PdfHandler->getSizeAndMetadata(TrivialMediaHandlerState, string)
#4 /srv/mediawiki/php-1.42.0-wmf.10/includes/utils/MWFileProps.php(89): MediaHandler->getSizeAndMetadataWithFallback(FSFile, string)
#5 /srv/mediawiki/php-1.42.0-wmf.10/includes/upload/UploadBase.php(546): MWFileProps->getPropsFromPath(string, string)
#6 /srv/mediawiki/php-1.42.0-wmf.10/includes/upload/UploadBase.php(478): UploadBase->verifyPartialFile()
#7 /srv/mediawiki/php-1.42.0-wmf.10/includes/upload/UploadBase.php(401): UploadBase->verifyFile()
#8 /srv/mediawiki/php-1.42.0-wmf.10/includes/jobqueue/jobs/PublishStashedFileJob.php(66): UploadBase->verifyUpload()
#9 /srv/mediawiki/php-1.42.0-wmf.10/extensions/EventBus/includes/JobExecutor.php(80): PublishStashedFileJob->run()
#10 /srv/mediawiki/rpc/RunSingleJob.php(60): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#11 {main}

Change 984865 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: move PublishStashedFile back to metal temporarily

https://gerrit.wikimedia.org/r/984865

Change 984865 had a related patch set uploaded (by Clément Goubert; author: Clément Goubert):

[operations/deployment-charts@master] changeprop-jobqueue: move PublishStashedFile back to metal temporarily

https://gerrit.wikimedia.org/r/984865

In T352515#9415206, @Clement_Goubert wrote:

We've moved the affected job (AssembleUploadChunks) back to metal temporarily.

There are still some of these errors popping up for frwikisource Special:Téléverser (Special:Upload), but not on other wikis as far as I can tell.

In T352515#9407128, @brennen wrote:

Seeing this also for PdfHandler:

Error

• service.version: 1.42.0-wmf.9
• trace.id: 9423484c-e79c-4e68-ad21-171f0d91a1ee
• Find trace.id in Logstash

labels.normalized_message

[{reqId}] {exception_url}   RuntimeException: firejail is enabled, but cannot be found

error.stack_trace

from /srv/mediawiki/php-1.42.0-wmf.9/includes/shell/CommandFactory.php(125)
#0 /srv/mediawiki/php-1.42.0-wmf.9/includes/shell/CommandFactory.php(189): MediaWiki\Shell\CommandFactory->getLocalShellboxOptions()
#1 /srv/mediawiki/php-1.42.0-wmf.9/extensions/PdfHandler/includes/PdfImage.php(110): MediaWiki\Shell\CommandFactory->createBoxed(string)
#2 /srv/mediawiki/php-1.42.0-wmf.9/extensions/PdfHandler/includes/PdfHandler.php(280): MediaWiki\Extension\PdfHandler\PdfImage->retrieveMetaData()
#3 /srv/mediawiki/php-1.42.0-wmf.9/includes/media/MediaHandler.php(227): MediaWiki\Extension\PdfHandler\PdfHandler->getSizeAndMetadata(TrivialMediaHandlerState, string)
#4 /srv/mediawiki/php-1.42.0-wmf.9/includes/utils/MWFileProps.php(89): MediaHandler->getSizeAndMetadataWithFallback(FSFile, string)
#5 /srv/mediawiki/php-1.42.0-wmf.9/includes/upload/UploadBase.php(546): MWFileProps->getPropsFromPath(string, string)
#6 /srv/mediawiki/php-1.42.0-wmf.9/includes/upload/UploadBase.php(478): UploadBase->verifyPartialFile()
#7 /srv/mediawiki/php-1.42.0-wmf.9/includes/upload/UploadBase.php(401): UploadBase->verifyFile()
#8 /srv/mediawiki/php-1.42.0-wmf.9/includes/upload/UploadFromFile.php(99): UploadBase->verifyUpload()
#9 /srv/mediawiki/php-1.42.0-wmf.9/includes/upload/UploadFromChunks.php(183): UploadFromFile->verifyUpload()
#10 /srv/mediawiki/php-1.42.0-wmf.9/includes/jobqueue/jobs/AssembleUploadChunksJob.php(67): UploadFromChunks->concatenateChunks()
#11 /srv/mediawiki/php-1.42.0-wmf.9/extensions/EventBus/includes/JobExecutor.php(80): AssembleUploadChunksJob->run()
#12 /srv/mediawiki/rpc/RunSingleJob.php(67): MediaWiki\Extension\EventBus\JobExecutor->execute(array)
#13 {main}

Change 984140 merged by jenkins-bot:

[operations/deployment-charts@master] changeprop-jobqueue: move AssembleUploadChunks back to metal temporarily

https://gerrit.wikimedia.org/r/984140

Change 984140 had a related patch set uploaded (by Hnowlan; author: Hnowlan):

[operations/deployment-charts@master] changeprop-jobqueue: move AssembleUploadChunks back to metal temporarily

https://gerrit.wikimedia.org/r/984140

Seeing this also for PdfHandler:

Change 979483 merged by jenkins-bot:

[mediawiki/core@master] Clean up DjVuImage::retrieveMetaData including shellbox support

https://gerrit.wikimedia.org/r/979483

Change 979483 had a related patch set uploaded (by Tim Starling; author: Tim Starling):

[mediawiki/core@master] Clean up DjVuImage::retrieveMetaData including shellbox support

https://gerrit.wikimedia.org/r/979483

Please wait until my forthcoming cleanup patch is merged before deploying this.

Change 979386 merged by jenkins-bot:

[mediawiki/core@master] media: Port DjVuImage::retrieveMetaData() to use BoxedCommand

https://gerrit.wikimedia.org/r/979386

Change 979386 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):

[mediawiki/core@master] media: Port DjVuImage::retrieveMetaData() to use BoxedCommand

https://gerrit.wikimedia.org/r/979386

Adding Structured-Data-Backlog as per https://www.mediawiki.org/wiki/Developers/Maintainers

In T352515#9374813, @Joe wrote:

According to logstash, this has happened 9 times in the last month, all on wikisource. So it is a user-facing problem but probably not as severe as I feared.

According to logstash, this has happened 9 times in the last month, all on wikisource. So it is a user-facing problem but probably not as severe as I feared.

So the context is that there was a problem on CLI when running maint scripts with the move of DjVu to shellbox. One quick stopgap would be to reintroduce the code in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/720143 under a feature flag, and flip it to true only when PHP_SAPI is not cli in mediawiki-config for now, while a proper fix for the issue is found.

Changing to UBN! because even if the frequency is low enough right now, this is a user-visible error.

The problem is apparently https://phabricator.wikimedia.org/T289228#7588412