@kevinbazira, all rounds have been created, they are sub-tasks of the current one. Please proceed on model training as you can! :)

@Trizek-WMF, thank you for creating all the rounds. I am working on generating datasets and models round by round and will be sharing updates on the sub-tasks.

The task generation for the Add Link task pool runs on one thread per DB server group (s1-s7). The two large DB server groups (in terms of number of wikis) are s5 (21 Wikipedias) and s3 (280 Wikipedias). For mid-sized wikis task generation took about 5-6 hours for a single wiki. It will probably take the same time for large wikis as well (we are going for a constant number of wikis, regardless of size); might take less time for small wikis where candidates can be exhausted before the required number of tasks are found. But assuming it doesn't, 280 Wikipedias is ~70 days, so we should probably run 4 instances of the refresh script in parallel on s3.

The script creates a lock with the ID of the wiki it is processing, so running multiple instances in parallel on the same dblist should be fine - the second thread will just skip the wiki that the first is already processing.

Proposal for streamlining the completion of remaining rounds:

1. Train models, verify models, publish datasets for all remaining wikis. This involves work from Machine-Learning-Team (cc @kevinbazira) and Research (cc @MGerlach). Growth doesn't need to be consulted on this phase; once these teams are happy with the models, the datasets can be published.
2. Growth engineers will populate the excluded section titles for all remaining wikis
3. Growth engineers will enable the backend for all remaining wikis, so the task pools begin to fill up
4. @Trizek-WMF can then verify the hasrecommendation:link results and check the API https://api.wikimedia.org/service/linkrecommendation/apidocs/#/default/get_v1_linkrecommendations__project___domain___page_title_
5. @Trizek-WMF can inform communities, and Growth engineers can enable the front-end, either staggered or en masse, depending on what works better for @Trizek-WMF

tl;dr I think we would all save time with context switching if we can do the model training, section title population, and backend enabling all at once, rather than in phases over a period of months. Then the actual presentation to communities could be done in a more staggered way if that is better from a community relations standpoint.

@KStoller-WMF @kevinbazira @MGerlach what do you think?

@kostajh +1 on doing the initial phases at once to avoid context switching.

The ML team will proceed with training models, evaluating them, and publishing datasets for all the remaining rounds.

As far as we keep a staggered deployment, I'm fine. :)

+1
This sounds like a great approach!

In T304110#8531763, @Trizek-WMF wrote:

When all wikis will have trained models, then we will resume deployments.

@kevinbazira
Do we have a rough ETA for when model training will be done for all Wikipedias? Thanks!

@KStoller-WMF, we are currently working on the 9th out of 18 rounds of wikis. Each of the 9 remaining rounds has ~20 models. ETA to train, evaluate, and publish all these models is about a month or more depending on the size of each of these wikis and whether or not we have to fine-tune the link recommendation algorithm to support a wiki's language-specific characters.

Will be sharing progress updates on the sub-tasks.

@Trizek-WMF Please further describe in this Phab ticket, what level of involvement is needed from CRS

On May 21, the last set of active wikis will have Add a link deployed. The next steps will be:

• T309263: Support languages whose add-a-link models were not published
• T308146: Integrate the model training and the deployment of "Add a link" to new Wikipedias exiting the Incubator

As a reminder, this deployment project started with the successful deployment of round 2, on Aug 14 2021 (round 1 being Growth pilot wikis).