I had missed that this had been tackled.
The new numbers make a lot morse sense to me. Thank you for the work @Milimetric !

I just deployed to the production cluster.
I will be following the executions of webrequest loading in case there are issues.
Otherwise, this is done :-)

I deployed the changes to the test cluster.
I will wait until tomorrow, to make sure there are no unexpected issues,
and then will deploy to the production cluster.

Mentioned in SAL (#wikimedia-analytics) [2024-08-19T20:45:03Z] <mforns> deployed airflow-dags to analytics_test instance for T368303

So to confirm, this means:

• over 1% of page views (after deducting known bots and spiders) are coming from clients with user agents that are entirely unknown to ua-parser. That is, the "Other" is already there in the raw wmf.webrequest_text dataset, and we've not created or normalized anything else to "Other".
• 0.26% is "Redacted" where we replace/normalize/summarise for privacy reasons browser/OS names in our pipeline.

Just in case it helps, I would like to add that we are very interested on automating these integration tests. We already created a ticket to explore about it (T371922: MPIC: Automate integration tests). At this time we have to run them manually because we need a database but it seems that GitLab services could run a database as a container within the pipeline (is that feature supported by our current GitLab installation?). That way these tests could be run automatically every time we push/merge something.
In fact this is something that we also miss when working with the APIs and the existing test suite we have for them. If I'm not wrong we already explored this feature there and the result of that exploration was that this is something not supported by the pipeline we have there (Gerrit + Jenkins).

Thank you @mforns

In T342267#10065753, @Milimetric wrote:

In T342267#10060142, @Krinkle wrote:

Hm.. it seems the "Other" bucket has grown slightly larger than our predictions of 0.26% prediction at T342267#9998984

[…] The "Other" that we're seeing now is just coming from UA parser actually identifying the stuff as "Other". […]

The thing that we figured would be ~ 0.26% in total would be all the items with "Redacted" in every dimension added together. […]

To-dos have been opened as subtasks of T329506: User Facing Metrics Platform Documentation

thanks @zeljkofilipin ! i will pop by in QTE's next office hours on Monday