Declaring this resolved.

I think using scap train makes it hard/impossible to do this. If you do use scap train to advance from (for example) group0 to group2, group1 will be promoted too.

In T347982#10261422, @CodeReviewBot wrote:

legoktm opened https://gitlab.wikimedia.org/repos/releng/scap/-/merge_requests/507

update-interwiki-cache: Give some output on success

In T375659#10259623, @dancy wrote:

Train is blocked at group1 due to T378108.

In T378108#10259626, @Pppery wrote:

I don't think it will get any worse - FundraiserLandingPage only runs on DonateWiki which is group1.

Train is blocked at group1 due to T378108.

I made this a train blocker because I'm worried about the warning log rate when rolling to group2.

Could this be related to T378003?

I wonder if this is related to T377912 which also mentions \Wikimedia\Message\ScalarParam

@Esanders Is a cherry-pick of https://gerrit.wikimedia.org/r/1082472 to wmf/1.43.0-wmf.28 sufficient to unblock the train?

In T377860#10250810, @Lucas_Werkmeister_WMDE wrote:

Pretty sure this is the same as T373087, which should be fixed with this week’s train.

In T377672#10247106, @dancy wrote:

@Esanders Hello. I'm on train duty this wek and @Jdlrobson added this task as a train blocker. Can you ensure that the relevant fix is backported to the wmf/1.43.0-wmf.28 branch?

@Esanders Hello. I'm on train duty this wek and @Jdlrobson added this task as a train blocker. Can you ensure that the relevant fix is backported to the wmf/1.43.0-wmf.28 branch?

Fixed in scap 4.111.0, which I deployed today.

Fixed in scap 4.111.0, which I deployed today.

Congrats @hashar! You're a CI hero.

In T376932#10219246, @egardner wrote:

Hey @dancy – what do you think about showing only the last X number of jobs on the main page and then having a "view all jobs" link to a dedicated jobs/ page that lists all the jobs for all time? Both could use the Codex table component, but this might be a way to keep the main page ( the / route ) cleaner.

Here are screenshots of the current prototype:

Previously docker commands were only run by the mwbuilder user (via sudo) during image build. Now docker is executed by whoever is running scap. The users in the docker group are a subset of the users in the deployment group, which is why this error happened.

In T374880#10153611, @dancy wrote:

Considering this resolved after https://gerrit.wikimedia.org/r/1073286. However I do have some remaining suspicions about the rollback behavior should scap deploy fail. I don't get the impression that the checks run on rollback, so a rollback isn't really a rollback for phatality.

Deployed with scap 4.104.0

https://chromewebstore.google.com/detail/wikimediadebug/binmakecefompkjggiklgjenddjoifbb?hl=en-US&pli=1 is showing version 2.9.0 so I think that's done too.

The updated Firefox add-on is available at https://addons.mozilla.org/en-US/firefox/addon/wikimedia-debug-header/.
The update has been uploaded to Chrome Web Store and is pending processing.

Considering this resolved after https://gerrit.wikimedia.org/r/1073286. However I do have some remaining suspicions about the rollback behavior should scap deploy fail. I don't get the impression that the checks run on rollback, so a rollback isn't really a rollback for phatality.

I filed https://github.com/moby/buildkit/issues/5329 about the build problem.

In T374478#10139303, @brennen wrote:

error: failed to solve: failed to load cache key: failed to fetch remote https://github.com/containers/dnsname.git: git stderr:
fatal: unable to access 'https://github.com/containers/dnsname.git/': Failed to connect to github.com port 443 after 129806 ms: Couldn't connect to server
: exit status 128

Unclear whether this is just a transient failure or if something else is up.

https://gerrit.wikimedia.org/r/1071027 failed on testservers when accessing https://commons.wikimedia.org/wiki/Campaign:Photographier_le_patrimoine_vivant_en_France_2024:

The target for today was to reach group1, but we only made it to group0 this afternoon. Error logs look ok right now but I want to let it marinate for a while. @hashar If you can, please roll the train to group1 during the next window. Thanks in advance!

https://gerrit.wikimedia.org/r/1070657 has been fully deployed.

Rolled the train back due to T374046

Possibly related to T373920 ?

In T373640#10117851, @hashar wrote:

@dancy I ran a couple backports:

• T373921 was confirmed to be verified since we had a repro url on test.wikidata.org which runs this week 1.43.0-wmf.21
• T373920 I have completed the deploy, but that needs real traffic to confirm the resolution

That leaves T373925 as the last blocker.

The warning rate ramped up a lot (~860/15 minutes) after rolling 1.43.0-wmf.21 to group0 today, so I'm making this task a train blocker.

Thanks @thiemowmde. So this issue is no longer a train blocker?

Will someone be backporting https://gerrit.wikimedia.org/r/1070232 to wmf/1.43.0-wmf.21 ?

https://gerrit.wikimedia.org/r/1067997 was included in this week's train branch (wmf/1.43.0-wmf.21) but I still see many Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value and Stats: Cannot associate label keys with label values: Not all initialized labels have an assigned value warnings being logged for testwikidatawiki or test.wikidata.org today.

Reproduced in train-dev.