A full write up with details of the 11 languages using Indic scripts (Marathi, Burmese, Malayalam, Telugu, Sinhala, Kannada, Gujarati, Nepali, Assamese, Punjabi, and Odia) that are configured in this last patch is on Mediawiki.

The updates look excellent to me!

Apparently this should have been an (oxymoronic) mini-epic.

I wanted to move this out of my ever-languishing 10% project pile.

Full Part 4—Refactoring & Analysis Notes are on Mediawiki.

Full write up on Mediawiki. In summary:

Thanks!

This ticket seems to be about how lsearchd parses wikitext, which isn't really relevant anymore. I can't reproduce the unwanted parsing behavior with current on-wiki search, so I'm closing the ticket. If there is still a problem, please re-open with new examples, or open a new ticket.

@EBernhardson, your write-up looks good to me!

This got done as part of T283366: Unpack Basque, Catalan, Danish Elasticsearch Analyzers, and I even noted that Basque had an extra boost from the kind of issues this specific ticket was supposed to address.

I'm going to close this ticket because the problem is resolved for a lot of non-punctuation characters, thanks to changes to the Elastic tokenizers.

I'm going to close this ticket because all of the example queries now do reasonable things (or, the unreasonable parts have to do with regex searches timing out, not Unicode characters).

This has been rolling around in my head for a while and something related came up today, so I wanted to jot down some notes to my future self, or to anyone else who may work on this.

Hi @34DSSDS. It is not an easy thing to manage individual search results. There are obvious knobs to turn to get the desired result here for this specific query, but it would be a big change and it would alter a lot of other suggestion results. Given that our current algorithm and configuration have been optimized to give the best overall results we could measure, dramatically changing it for one query will almost certainly lead to worse results in general.

Other than @IKhitron's suggestion to add the main page to the list when suggesting the sub-page, is anything here that isn't covered by T159861: Add an is_subpage field to elasticsearch documents and use as a scoring feature, which explicitly takes the sub-page status into account? Should we merge the tickets?

I looked at all of these as best I could, and I decided on a general mapping to standard Arabic forms internally. Arabic does that to some degree, as does Persian! And for the languages without custom stemmers and stop word filters, the character used internally doesn't matter, as long as the desired words can find each other.

In reading up on the ligatures, I found another ligature (yod-yod-patah ײַ) that has several variants, one using a ligature from above (double-yod + patah ײַ), one with separate characters (yod + yod + patah ייַ), and a less common variant with the patah in the middle (yod + patah + yod יַי). It looks like icu_normalizer already converts the single-character form (ײַ) to one using the double-yod ligature (ײַ).

While we had planned to expand the deployment of the hiragana-to-katakana mapping from English to most other languages (though not Japanese), testing revealed that doing the mapping pre-tokenization interfered with the new ICU tokenizer's ability to parse Japanese text (on non-Japanese wikis).

In T362442#9714689, @Xaosflux wrote:

#REDIRECT [[Page#Anchor]] - going to it should still lead you to the Anchor, not to #top of that page - as is being done here.

Showing the canonical page title in the suggestion was a design decision that was made for Vector 2022, though it can be confusing when the full title isn't obviously related to the redirect title. T303013 has some potential heuristics for deciding when to show the redirect info. Feel free to chime in over there if you don't feel like your use case would be covered (the specific artificial example here would be covered).

Is this substantially different from T331389? ("...<nowiki> output in search result descriptions")

In T358495#9705136, @NMW03 wrote:

Not sure if this task fixes that, lowercasing I and dotted I (İ) returns different lowercase letters

Not getting automated tags for some reason, but this is included in 1.42.0-wmf.25, so it will be deployed soon.

Full write-up (and it's a lot!) is on MediaWiki.

The full write up is on MediaWiki.

This is done for commons and wikidata for the production clusters (eqiad and codfw) as a result of T342444. (wikidata hasn't reindexed in cloudelastic yet, but it is in the queue.)