A quick note - I tried logging in/logging out between two versions - docker-registry.wikimedia.org/dev/buster-php74-fpm:1.0.0-s3 and docker-registry.wikimedia.org/dev/buster-php81-fpm:1.0.1-s2 -> and the keys worked. EG - the key created under 7.4 worked when trying to log in on 8.1 -> and the same vice versa - 2FA created on 8.1 worked on 7.4.

I got fixated on the composer issue and spend most of my time trying to rebuild composer/etc. But I definitely see a valid point in getting keys set up on 7.4, then bump to 8.1 and try to log in again -> I'll try to do it right now - plus for 7.4 I'll fetch the mediawiki/vendor

I don't think it's a problem anymore. I tried to find out what could fix it but at first glance I don't see it. At this moment - I'm at:

• MediaWiki core - latest master branch
• OATHAuth and WebAuthn - latest branch

On docker I'm running PHP 8.1.20 - composer install/update scripts worth without errors - I removed all vendor folders across the app.

~/projects/mediawiki(master○) » docker compose exec mediawiki php -v         
WARN[0000] /Users/pmiazga/projects/wmf/mediawiki/docker-compose.override.yml: `version` is obsolete 
PHP 8.1.20 (cli) (built: Jun  9 2023 07:40:35) (NTS)
Copyright (c) The PHP Group
Zend Engine v4.1.20, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.20, Copyright (c), by Zend Technologies
    with Xdebug v3.2.1, Copyright (c) 2002-2023, by Derick Rethans

composer update output:

~/projects/mediawiki(master○) » docker compose exec mediawiki composer update
> MediaWiki\Composer\VersionChecker::onEvent
Loading composer repositories with package information
Updating dependencies                                 
Nothing to modify in lock file
Installing dependencies from lock file (including require-dev)
Package operations: 0 installs, 0 updates, 0 removals
Package fgrosse/phpasn1 is abandoned, you should avoid using it. No replacement was suggested.
Package web-auth/metadata-service is abandoned, you should avoid using it. Use web-auth/webauthn-lib instead.
Generating optimized autoload files
70 packages you are using are looking for funding.
Use the `composer fund` command to find out more!
> MediaWiki\Composer\ComposerVendorHtaccessCreator::onEvent

Work is done, the difference between Graphite and Prometheus stats is caused by how those process stats.

The PR @Tgr introduced made me wonder again - can we have a place where can store request-based state? We decided not to keep any state in MediaWikiServices (T218555). We have to god class RequestContext - and every time I see it, I want to get rid of it.

Dashboards are updated.

I'm tagging observability as they could be interested in this topic. I would like to look into it, and I'll push some POC soon.

I need to update dashboards on our side, and then I'll close this ticket.

Unassigning myself as I'm not actively working on it and I don't want to give a false impression. Huge kudos to @ArthurTaylor and @kostajh for driving it.

Looks like error is gone.

so one more prometheus label domain: ["central"|"local"]?

do we have any updates on this matter? I see the last log from yesterday

Ok, quick update, I followed the code little bit more, looks like may be a limitation somewhere. From the code I see that samples do not contain labels as a map/associative array - it's a list of values.

It looks like it's an arbitrary limitation of the library. The reason why this happens is the fact that stats buffering library is stateful. The StatsFactory::getCounter that calls ::getMetric() uses cache to buffer all metrics.

In T377035#10238323, @daniel wrote:

@pmiazga is it possible you saw T375615?

Looks like this is still happening in T375658

In T377035#10238121, @daniel wrote:

@pmiazga How did you find that database error?

Updated ticket title to match the current state of things. yesterday we merged the minimal OTEL tracing library. Thank you @mszabo for great work!.

This is already possible and we're kinda already doing it by injecting custom Monolog Processors - for example we add WikiProcessor (https://gerrit.wikimedia.org/g/operations/mediawiki-config/+/e976dcf9860096df1c81a891947d886fe49f7eec/wmf-config/logging.php#117)

I wonder if we want to inject the Monolog processor to the entire stack. We could do something like that:

One of the reasons why we might need to support HTTP mode is the TLS overhead (performance cost) but I don't think it justifies the additional complexity we need to maintain. I wonder if we have any use cases or requirements to support mixed mode? Only thing that comes to my mind is dev environments - but those are just HTTP mode (without SSL).

@kostajh yes, I think you can get those rates from this work. Once I get a green light I'll implement this solution. I expect to land it by the end of this week.

After a quick analysis of our logging system:

Where this job could log in when it fails? Maybe it could push messages to mediawiki-core-bots channel for better visibility?

@kostajh - one question that got me thinking - I assume that expiryAfterDays won't be changed often, but it probably won't change at all. Do we foresee any scenarios when this value gets changed on prod? If we increase it - is it going to "re-enable" all temp-accounts that were previously expired but are still "active" ?

This sounds good to me. Thank you for handling this. @kostajh most likely this ticket should be assigned to you and @ArthurTaylor were the drivers of this work.

Moving to the current sprint as the work is almost done. If we need further changes, then please move it to Soon.

@kostajh it's not a blocker for IP Masking deployment. We need it to monitor account creations to catch possible misbehavior of the system. We will pick it up soon.

Moving to current sprint as we're currently investigating login issues ( T372702 )

Moving to current sprint as this is could be related to the ticket we're currently working on - T372702.

could be caused by recent Account-Vanishing changes. @Seddon could you investigate this issue?

@kostajh moving to radar - in case you need our help please ping us.

@matmarex also tried to reproduce without too much luck. This may be related to T372702 but we're not sure.

I cannot reproduce this behavior.

@Krinkle do you have any thoughts on this one ?

I would say that in our architecture, Skin class is hidden somewhere deep and it is not last thing to call, it's usually called by some EntryPoint, that calls OutputPage, which later on calls the Skin, and then on the return -> we still allow some modifications. In other words -> Skin::outputPage() is not the last thing that happens, therefore I'm not sure if this is supposed to echo stuff.

There is a quick way already - Telemetry::getRequestHeaders().

Regression is caused by https://gerrit.wikimedia.org/r/c/mediawiki/libs/less.php/+/1013381

I agree with @Krinkle on the sizing -> I was surprised how much code the PHP open-telemetry introduces. It's important to notice that php-otel provides not only traces, but also metrics and logging functionality. We would need only tracing capabilities but PHP OTel SDK comes as bundle.
We could skip the protobuf extension and send data in json format - slower and requires more throughput. The google/protobuf lib is not recommended:

The native protobuf library is significantly slower than the extension. We strongly encourage the use of the extension.

For exporter we could use Guzzle which is already part of MediaWiki - but we would require all HTTP messaging/factory interfaces from PSR and I don't think we have it.

@taavi could you check this issue? Looks like it's related to our previous 2FA work. I'll be happy to review it.

Thank you for tagging this task with good first task for Wikimedia newcomers!