@Urbanecm subscribed, tyvm!

Okay, the script has been updated to deal with account creations by other performers, if the IP/User Agent for those performers can be found, and to skip creation of accounts for any where the info is no longer available.

@JJMC89 Can I get your opinion on the above (Urbanecm's comment)? I want to make sure we are meeting everyone's needs here before we roll this out. Thanks!

The above two patches, or something like them, might be able to handle the case of forced user creations by others. Leaving them here for comment and discussion.

In T378401#10307273, @Tgr wrote:

...

I don't think it's worth the effort though. I think it's fine to just let such cases fail and log the failure. If we want to be honest, no one ever looks at maintenance script error output anyway.

In T378401#10307169, @JJMC89 wrote:

In T378401#10303452, @ArielGlenn wrote:

there were several failures, because the IP address for the user had been globally blocked with reason 'no open proxies'.

How/where are these accounts getting created in the first place if the IP is blocked?

The remaining 35 or so got the IP of 127.0.0.1 and an empty user agent string, because those accounts had all been force created by an admin responding to https://en.wikipedia.org/wiki/Wikipedia:Request_an_account

Are you fetching the correct data? These should have real IPs/UAs from the person that created the account.

To skip over temp account names and not attemp to create them until they are enabled on metawiki and loginwiki, we'd need to update InitialiseSettings.php to set $wgAutoCreateTempUser['known'] to true for those wikis, so that temp names will be recognized (otherwise we get a blanket false response from the check). Is that okay?

One more comment:: do we want to check for global blocks in advance before trying to autocreate?

I ran with '2 hours ago' and it was fine. One user was autocreated with an appropriate IP and user agent.

Did some dry runs for loginwiki backfill. Results:

• for one day's worth, 10622 uids checked, 44 accounts would be created.
• for 7 days, 52582 uids checked, 232 accounts would be created, of which two were temporary accounts and would fail.
• for 2 hours, 709 uids checked and two accounts would be created.

All dry runs took less than ten minutes.

I have a patch ready but no point in pushing it up before the script lands via the train. Maybe I'll just claim this task in advance anyways. Swipe!

@Tgr --startdate "5 days ago" works as expected, you're all set already.

I have a question about the change made here https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/1073764

In T373789#10112786, @Jdforrester-WMF wrote:

I've worked around it for now by manually editing composer.json to pin version 1.1.4, but that is not ideal.

Which composer.json? MediaWiki provides psr/log, which is pinned to 1.1.4, and has been since we upgraded it in 2021.

core composer.json asks for 1.1.4. It may be that I ran composer incorrectly, but for whatever reason, the version in the checkuser vendors directory was a 3.x one.

Encountered when testing https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/1067385 for https://phabricator.wikimedia.org/T371267

In T373447#10096967, @matmarex wrote:

I thought it was intentional.

The patch currently up is missing the CheckUser related stuff, it's just bare bones.

In T371267#10023454, @Dreamy_Jazz wrote:

...

As to other mitigations we could:

• Fetch the correct IP, user agent, and timestamp by finding it on the wiki where the user was actually created. Then it would appear in CheckUser as if the account was autocreated normally.
• Add some kind of flag to indicate that this was created by a backfill script (such as a custom user agent string that says Backfill creation by maintenance script)

In T371267#10027610, @Niharika wrote:

@ArielGlenn Hi there! Hope you're doing well. We are still a bit far from deploying to any major production wikis (1-2 quarters possibly). It's hard to put any firm dates around this because of multiple dependencies on other teams and on Legal. We haven't decided on which wikis we will pilot on either.

Hey @Dreamy_Jazz can you point me to some sort of schedule as to the rollout? I'm not interested in hard dates, just getting a general sense of the order and time frame.

After discussion during the weekly team meeting, we decided that further work on this task is not warranted, so closing. Next steps will be in a separate task for doing the backfill.

I can find insertion metrics for local account creation in prometheus (though apparently not per wiki, so that's a problem), but I can't seem to find a topic for retries or failures. Sample link for that, so we have it: https://prometheus-eqiad.wikimedia.org/ops/graph?g0.expr=kafka_server_BrokerTopicMetrics_MessagesIn_total%7Btopic%3D%22eqiad.mediawiki.job.CentralAuthCreateLocalAccountJob%22%7D&g0.tab=1&g0.stacked=0&g0.show_exemplars=0&g0.range_input=1h&g0.end_input=2024-07-29%2012%3A00%3A20&g0.moment_input=2024-07-29%2012%3A00%3A20

Checking metawiki for the same accounts created on enwiki for the same uid range as earlier, there's about 620 that were globally registered in July that have no accounts on metawiki. So that's somewhat worse than loginwiki, though still a small number.

So, of the 140 accounts created on enwiki in the last 30 days, no entry on loginwiki, and having an entry in the globalusers table made in the last 30 days, about half of those were created on behalf of the user by an admin. I'll have a look at a good chunk of the rest to see if there's anything interesting about those other 70-ish accounts.

At least some of the problematic names are accounts that were created on behalf of a user by another user, via https://en.wikipedia.org/wiki/Wikipedia:Request_an_account/Guide (see code at https://github.com/enwikipedia-acc/waca/tree/rel7.17 )

There are three open tasks referencing some sort of failure with this job, no idea if any are relevant now, but listing them here just in case:
T220769: Account created without having a loginwiki or metawiki automatically created (2019)
T306636: DBQueryError: Error 1213: Deadlock found in UserOptionsManager::saveOptionsInternal (2022)
T173451: New accounts not being reliably created on loginwiki (2017)

Okay, I checked accounts created during the last approximately 30 days on enwiki. There were about 134700 users created; about 400 of those had no entry in the loginwiki user table, but they all had entries in the globaluser table (looking entries up on loginwiki and globaluser by name).

I've done some simple spot checks, looking at recent account creation on en wiki vs accounts on loginwiki, and it's good so far, but that's small amounts of data. Working on getting a more comprehensive check for say a 30 day period, we'll see what it shows.

Hey Daniel, I'd just assumed that getting added to the analytics-privatedata-users group would be redundant so thanks for catching that.

What are the next steps on this? Any information I need to provide, or anyone I should nudge?

In T348388#9916271, @Bugreporter wrote:

In T348388#9916155, @Tgr wrote:

Actually, I don't think unattached accounts can log in at all. That was disabled way back in rOMWC165ecbfaba66: Set $wgCentralAuthStrict = true;.

So it would be the time to remove support for unattached accounts (i.e. assume all accounts are mergable) in CentralAuth.

I don't mind if subscribers aren't copied, as long as there's an easy way for subscribers to the parent task to be notified about children created if they like, with a one click add or something. Basically, I don't want to have to actively monitor a pile of parent tasks to see if any child tasks of interest were created that I want also to subscribe to.

In T365260#9834222, @kostajh wrote:

...

@ArielGlenn I don't think there is much we can do about this. I don't think it's possible to avoid posting a comment back on Gerrit if the job fails. If the syntax errors in the patch are fixed, the Sonarcloud link will work.

Just seen again on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CentralAuth/+/1035722 with result 404 and

Claiming this for now, as I'm working on the alternative proposed in T354701#9753206

In T316303#9833459, @XtexChooser wrote:

It was saying I am globally IP blocked. idk why but now everything is fine and I am able to autocreate accounts. I created a couple of local accounts and it worked fine. Thanks!

In T316303#9833454, @XtexChooser wrote:

Not for me on en wikivoyage and wikiversity. But seems working on most small wikis.

Hey @bd808 are you in the position to check whether autocreate is now working for you from a blocked IP range, or did you work around this issue via stewards or whatnot?

In T364939#9826634, @Tgr wrote:

Based on the design review, would it make sense to also add an iframe-in-a-modal version?

On the one hand, for wikis which are on a single second-level domain, the cookie-handling-related benefits of a popup don't exist, and a modal is much nicer UX.

On the other hand, currently we disallow opening the login page in an iframe, even if the parent is same-origin; it can be used to elevate an XSS into password theft via clickjacking. (See T255881 and related tasks.)

Giving this back to Paladox, as he did most of the actual work. I just did a little tweak to that.

Looks ok in deployment-prep, so I'm just gonna close this.

With the latest patch, local testing renders a log entry for forced user creation as

What third party cookie testing etc flags might you have enabled or disabled? Anything possibly interfering from there? I took a fresh install of Chrome, did the steps described, and at step 4 I was still logged in on loginwiki as the second user. Returning to en.wp after that, I was still logged in there as well. Chrome version 124, linux, not logged into Google or my WMF account when using the browser. I was in an incognito window, and the Privacy and Security settings say "Third-party cookies are blocked in Incognito mode".

Reviving this old patch, let's see what happens.

The above patch fails CI for three tests not in the Database group. All three call User::isAllowed() at some point, which now would require database access. Two tests are in ConfirmEdit and one is in core (DumpableObjectsTest). I could add al lthree to @group Database or I could mock out CentralAuthHooks and have the onUserGetRights method always return true in those tests, or maybe there's some better approach. Poking @Tgr for advice or a pointer.

Note that for any of those issue links in the task description, I get "Access is denied to this issue". But the issue linked to in T359957#9625065 is visible at least.

In T358329#9580035, @Ladsgroup wrote:

Yes I agree but that is inside a public git repository (from what I'm seeing) which means it'll be confusing will be potentially removed again. That's why I didn't put it there. It needs a better solution.

In T358329#9579391, @Ladsgroup wrote:

I had to reclone it again from db11, it's usually better to set the db to read only before doing the cloning (read only if you cloning from master, "stop slave" if you're cloning from another replica.) even though binlog can be replayed, there are statements happening that wouldn't be written to binlogs. Regardless, that's fixed now.

Regarding password, it's just a user on master with that password, you can just change its password. I did exactly that, the password is in my home directory on db11 and only readable by root.

In T358329#9577327, @Ladsgroup wrote:

Hi 👋 I suggest stopping to touch this. I will take a look soon. Regarding databases, if you're not 100% sure what you're doing, you usually end up making things worse. Trust me, been there done that

The cloning procedure is done for db14 but we are currently hunting around for the replication password, not where the docs ( https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/Databases#Starting_Replication ) say it should be, not anywhere in that repo ever, apparently.

In the end @TheresNoTime figured it out: puppet was starting mariadb automatically when we didn't want it running and hence creating that file complained of in the error above. The cloning process looks like it's going ok at the moment.

Just to explain to folks who might be following along, what's happening: the primary server (db13) will be cloned (via mariabackup --innobackupex) to the new replica; a new instance is bring created now for that. While that is happening, replication will be stopped and the primary will remain read-only. I am guessing that this will be a matter of some hours rather than a day. More updates as available.

1 and 2 are both role dumps::generation::server::spare and have been so since at least last July. See https://gerrit.wikimedia.org/r/c/operations/puppet/+/936379 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/893265
While any nfs spare could in theory be swapped in for any production host, we have other newer spares with decent size filesystems for that; these are idle and can go at any time.

I'd prefer the beta be kept in the name, making it clear that these are wikis on the deployment cluster.

This is in progress still, notes so I don't forget:

• we run 7zs after the bz2 history files, that job will remain untouched
• need to adjust the script that does history backfil to not move files into place if there is a temp or bz2 file that appeared in the meantime (protection in case the fillin script and the main worker both reach the same files)
• could do md5s of these files as we go but let's see how much we gain without, it would be safer as far as avoiding overlaps

We may want to test the behaviour when going from logged in on a wiki on beta.wmflabs.org (let's say en.wikipedia) and then visiting some-language.some-wiki.beta.wmcloud.org which is not designated as the "representative wiki" for that wiki family, and see if the behaviour is different from visiting the "representative wiki" immediately after login on en.wp.beta.wmflabs.org. These scenarios behave differently for me in production.

As I try to help plan this out, I have accumulated some questions.

Listing more extensions not named above that appear in Special:Versions and seem to me to not be needed at loginwiki:

• 3D
• CodeEditor
• CodeMirror
• ElectronPdfService
• Global Usage
• Kartographer
• MediaModeration
• RevisionSlider
• TemplateStyles
• TextExtracts
• TwoColConflict

and maybe others.

Verified for Firefox: I logged out, deleted all *wik*org cookies except for non SUL sites (phab, etherpad and so on), set Enhanced Tracking Protection to "custom" and chose "All cross-site cookies" (see image below). After login at en.wikipedia, after the end of the redirect/session creation chain for commons.wikimedia.org, I have session cookies for commons with session id, UserId, UserName stored locally. When I go to visit commons,wm.o, these cookies are sent to the web server and I am logged in as a result.
Firefox version: 121.0, linux.

While T17294 seems stalled, is there anything do be done for CentralAuth in the meantime?

Sounds good to me, I'll lurk until input is called for :-)

The one thing I didn't think to check. Of course it works fine on other accounts. Closing!

In T205992#9413542, @Tgr wrote:

Seems to work fine for me.

The above patch went out with last week's train, and is on all wikis (1.42.0-wmf.9) but the behaviour is unchanged so I'll need to look into this further.

I wonder why I thought he was in mine? Wishful thinking maybe. Hope it all goes well!

We missed you, I'm guessing that you got notice of this too late to make the training? We can reschedule in any case.

I get confused by the name 'domain' every time. 'database' or 'dbname' is much clearer imo.