In T374729#10166978, @JMeybohm wrote:

Restarts are all done. You may give it another try @dcausse

In T374987#10162634, @EBernhardson wrote:

In T374987#10155932, @dcausse wrote:

In T374987#10154319, @Pppery wrote:

But also why is CirrusSearch updating on closed wikis anyway? There shouldn't be anything to update.

Closed wikis are still being served and search is available there, CirrusSearch is still scanning some documents to do some automatic cleanup which I suspect is the source of these API requests.

I wonder, should we turn that off for closed wikis? We are already doing filtering based on sitematrix there for private wikis in some cases, we could similarly exclude closed wikis.

Was looking at a dashboard that already filtered non-mediasearch poolcounter errors

In T374987#10154319, @Pppery wrote:

But also why is CirrusSearch updating on closed wikis anyway? There shouldn't be anything to update.

Closed wikis are still being served and search is available there, CirrusSearch is still scanning some documents to do some automatic cleanup which I suspect is the source of these API requests.

In the RFC I read

In some cases, the URI is specified as an IP address rather than a
hostname. In this case, the iPAddress subjectAltName must be present
in the certificate and must exactly match the IP in the URI.

Thanks for looking into this!
Now failing with javax.net.ssl.SSLHandshakeException: No subject alternative DNS name matching kafka-main-eqiad.external-services.svc.cluster.local found use_all_dns_ips and now it seems that it wants to validate the hostname passed via bootstrap.servers...
I'll investigate more to see if there are more options, if we fail to workaround this do you think it'll be acceptable to add kafka-main-eqiad.external-services.svc.cluster.local as a valid alternative in the cert?

not sure that the check_categories.py --ping is necessary and could be dropped imo, it should already be covered by some other sensors.

We could perhaps adapt modules/query_service/files/monitor/prometheus-blazegraph-exporter.py to take care of running this query by possibly re-using the same gauge blazegraph_lastupdated but adapting the query depending on the namespace it's running.

In T370665#10147571, @JAllemandou wrote:

I think the next steps of this analysis are to investigate why we have late-events at specific hours (my guts tell me to look at Event-gate glitches), and why some datasets have those late events as well (hint: 6 over 7 are mediawiki generated events).

Unsure if related but we recently found that some MW requests might last for several hours (T374662), so depending on how the event is created it's possible that late-events are created by MW:

• Event is created by setting meta.dt (https://gerrit.wikimedia.org/g/mediawiki/extensions/EventBus/+/79790fdff075c7fa0e4a359e6f359cb35ddfac36/includes/EventFactory.php#354)
• A DeferredUpdate is used to delay the push to EventGate (see for instance revision-create: https://gerrit.wikimedia.org/g/mediawiki/extensions/EventBus/+/79790fdff075c7fa0e4a359e6f359cb35ddfac36/includes/EventBusHooks.php#407)

I believe that this might possibly lead to late events being sent by MW.

In T330525#10142839, @Sj wrote:

@dcausse , thinking of your comment from office hours this month: this is the sort of hopefully-separable work that I imagine finding a grapher in residence to work on ;)

The current hypothesis is that the problem happen right after the node comes online and is advertised by the cluster as usable but that node is not yet allowed by the egress rule. The kafka-client is then too confused and the job enters a crash loop. Other other jobs seem to be more tolerant to this setup. To be precise the search job also suffered some blips during the process impacting the search update lag but I think this is totally acceptable.

In T331127#10141617, @dcausse wrote:

I think we still have issues with this, looking closer it appears that MW is not properly flagging this event as a page move but a page edit which then is misleading CirrusSearch into thinking that it's a "normal" edit: P69074.
@Ottomata is this expected? is this possible that page moves between namespaces are not properly identified by the page_change stream?

In T331127#10139495, @Bearcat wrote:

After having not seen this problem recur for several weeks since the last batch I posted about was cleared, there's been a new isolated case recurring again, with User:Citrivescence/Asher Perlman still not dropping from https://en.wikipedia.org/w/index.php?search=incategory%3A%22Living_people%22&title=Special%3ASearch&profile=advanced&fulltext=1&ns2=1&ns3=1&ns118=1&ns119=1 after being moved to mainspace five days ago. Given the "two weeks to work through all possible pages" comment above, I suppose it may take a while, but I'm just reporting it so that it's known.

@bking everything looks good, thanks!

I think that all patches have been merged, most of them deployed except one which should get deployed via the train tomorrow for group1 (re-enable fine-tuning per language).

In T368067#10130728, @dcausse wrote:

Untagging WDQS as I believe there are no issues with WDQS, I think the query does not work because https://www.wikidata.org/wiki/Q126787117 was created without a trailing slash for the P856 property.

Untagging WDQS as I believe there are no issues with WDQS, I think the query does not work because https://www.wikidata.org/wiki/Q126787117 was created without a trailing slash for the P856 property.

In T271776#10118149, @dcausse wrote:

the data seems to be indexed so it might be trivial to implement these keywords, moving to needs triage to raise visibility.

the data seems to be indexed so it might be trivial to implement these keywords, moving to needs triage to raise visibility.

Seems to be fixed, tentatively closing

In T373086#10116237, @gmodena wrote:

Looks like the spike is only related to CirrusSearch?

Unfortunately wdqs2021 is still consumer from the wrong topic after the transfer.
Looking closer it appears that the service definition for the wdqs-updater is duplicated in two locations:

• /etc/systemd/system/wdqs-updater.service containing the wrong topic codfw.rdf-streaming-updater.mutation
• /lib/systemd/system/wdqs-updater.service with the right topic codfw.rdf-streaming-updater.mutation-main

In T371401#10065479, @EBernhardson wrote:

https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1060433 appears to be not-needed in the end, I don't see where we use these manually tuned profiles

After some review it turns out the code that used the language-tuned profiles was lost as part of splitting WikibaseCirrusSearch out of the Wikibase repo. All the related machinery still exists and it would be pretty easy to add it back in now, but I wonder if we should be doing testing of some sort to verify those profiles are better than the defaults we've been using. They were at the time, but user behaviour can change in the 5 years that have passed.

My understanding is that https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseCirrusSearch/+/1064792 is going to fix both this ticket and T371929.
From this ticket description it is not entirely clear if the ask is also to index the full P18 statements or just the flag that indicates the presence of the use of this property, for the former I'm with Erik this possibly adds a lot of new tokens that might be be particularly hard to search for (untokenized URLs) and thus probably not very useful.

Should be resolved with https://gerrit.wikimedia.org/r/c/mediawiki/extensions/WikibaseCirrusSearch/+/1064792

Interesting, I thought as well that the 1k limits would apply to nested bool queries (which is probably one reason it was set to 256 initially). It means that we can probably safely bump the limit to 1k without even nesting bool queries. I'm not clear why it has such an impact when getting past 2.5k and I have no clue if a terms query would perform significantly better, it's less costly for sure since there's no need to analyze & rewrite the query, we could probably test this as well to see the impact?
So perhaps we can at least bump to 1k right now with a simple config change and ponder what to do next based on some testing of the terms query? If the terms query does not show a significant gain compared to nested bool queries we might just use this?

I was able to trigger the backfill for wikidatawiki_content running another re-index for an unrelated wiki on both eqiad and codfw, seems to me that there's an early stop when all the re-index are done, it should perhaps double check that no remaining backfills are needed before quitting?

Current status:

• wikidata has been re-indexed
• haslabel:mul should work
• search autocomplete should use the mul label when it's part of the fallback chain
• full text search results might benefit from better recall after merging https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1060433 (this one might or might not need commons to be re-indexed first)
• https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1060433 appears to be not-needed in the end, I don't see where we use these manually tuned profiles

This should work now, we had to re-index to wikidata

~ is definitely a confusing character for search and is heavily overloaded:

• used to force entering the search results page when used as a prefix
• considered as a punctuation and ignored by many text analysis components
• used to trigger fuzziness word~
• used to control the phrase slop in "foo bar"~2
• used to perform a phrase search on stems "foos bars"~
• has some restrictions on page titles (impossible to create a page named ~~~)

Regarding fallbacks WikibaseCirrusSearch is relying on \Wikibase\Lib\TermLanguageFallbackChain::getFetchLanguageCodes, the order in which these languages are returned is quite important as well as the weight attributed to such matches are inversely proportional to its position in this array.

The procedure should be:

• merge & deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1060430
• reindex wikibase wikis: wikidata, testwikidata, commonswiki (at this point search should be already better in Special:Search and T371352 should be fixed)
• merge & deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1060433 (which should slightly increase recall in some cases when using Special:Search)
• merge & deploy https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/1060449 should start using the mul fallback for manually tuned profiles (en, fr, de, es)

The mul labels and descriptions (can we have mul descriptions?) are currently not indexed and explains to some degree why search is behaving poorly on these items. We'll index those and see how it performs, tuning search might come as a separate step.
If I'm not mistaken mul is considered a fallback for all languages so it should always be queried.

Dashboard is up at https://grafana.wikimedia.org/d/8xDerelVz/search-update-lag-slo?orgId=1
A patch to grafana-grilly is upload at https://gerrit.wikimedia.org/r/c/operations/grafana-grizzly/+/1060150 (but not sure how to move forward with it)

In T371746#10045764, @Pppery wrote:

Why doesn't whatever logic allows "Double tilde" to be found work here then, though? I was assuming that was found via the redirect "~~", which you seem to be implying should have the same problem.

In T371746#10045635, @Pppery wrote:

I created the redirect "~~ (album)". It does show up in the completion index for "~~~" (after 5 irrelevant results), but does not show up in the search index. Odd.

Unchecked the prerequisite regarding kafka topics, the split graph hosts are currently configured to consume from the full graph topic, the reload should not start (probably be stopped/restarted on wdqs1021) before https://gerrit.wikimedia.org/r/c/operations/puppet/+/1060049 is merged and applied to the corresponding hosts.

As part of the work to expose two new endpoints serving the split graph (T364363) we are configuring new wdqs hosts to run blazegraph.
The way maxlag is propagated from WDQS to mediawiki is by measuring the most lagged wdqs host that is online.
In order to know what wdqs hosts are online we measure the number of queries that it serves.
We also run some "monitoring queries" internally to measure the health of the system and in order for these monitoring queries to not interfere with this system we flag such internal user-agents so that they're ignored.
This is where we made a mistake, meaning that a monitoring user-agent was not properly flagged as internal and caused a new host (not yet fully loaded) to be considered online and thus taken into account by maxlag.

In T371746#10042876, @Pppery wrote:

I was unaware of the meaning of "~" as a search operator. Is that documented somewhere?

In T371746#10042876, @Pppery wrote:

Does this mean that if I were to create a redirect "~~ (album)" -> Tilde Tilde Tilde for example then it would be found by that search. That's hacky but tolerable.

Search queries prefixed with ~ has a special meaning for Special:Search, it instructs the UI to go to Special:Search rather than the article page if it exists, it's the reason why ~~ is found when searching ~~~.
This is sadly not the sole reason why it's not found, ~ are likely ignored in the fulltext search index and thus only relying on titles or redirects to find ~~. Given that there's no way to add such titles nor redirects with ~~~ I don't see an easy way to solve this issue because search needs to pull this data from somewhere.

In T356161#10041159, @Nikki wrote:

In T356161#9497847, @Lucas_Werkmeister_WMDE wrote:

I doubt the same situation is possible on Wikidata, since we disallow moving items, properties or lexemes, and the moving seems to be a crucial part of how the history was split here.

It seems it is, the same problem was reported on Wikidata a few days ago - https://www.wikidata.org/wiki/Wikidata:Report_a_technical_problem#Non-unique_statement_id_in_Q85046372

In T356161#10040266, @Nikki wrote:

Is this the same as T317530?