In T380009#10329055, @revi wrote:

In T380009#10328760, @Platonides wrote:

In T380009#10326074, @eoghan wrote:

So the issue is coming from the vrts_aliases.py cron job. Something has changed in how gmail is responding to emails here and claiming they're valid. I'm going to try see if we can change that to ignore the gmail check

OIT?

Office IT, which is renamed to IT Services, is WMF dept handling employee account and their inboxes (which are handled by Google mailbox). (EDIT: I know this from my past employment at WMF, and it is probably usually-unseen group for almost all volunteers.)

(I've created meta redirect for Office IT and OIT.)

We're in touch with ITS and will be doing follow up testing with them next week.

In T380009#10328760, @Platonides wrote:

In T380009#10326074, @eoghan wrote:

So the issue is coming from the vrts_aliases.py cron job. Something has changed in how gmail is responding to emails here and claiming they're valid. I'm going to try see if we can change that to ignore the gmail check

OIT?

In T380009#10326074, @eoghan wrote:

So the issue is coming from the vrts_aliases.py cron job. Something has changed in how gmail is responding to emails here and claiming they're valid. I'm going to try see if we can change that to ignore the gmail check

We've made a change to the aliases routing script which we believe has fixed the problem. I've verified that mail is delivering to vrts now, and we've seen two of our test tickets arrive.

Change #1091628 merged by EoghanGaffney:

[operations/puppet@production] mx: Update vrts_aliases script not to check gmail for wm.o addresses

https://gerrit.wikimedia.org/r/1091628

Change #1091628 had a related patch set uploaded (by EoghanGaffney; author: EoghanGaffney):

[operations/puppet@production] mx: Update vrts_aliases script not to check gmail for wm.o addresses

https://gerrit.wikimedia.org/r/1091628

So the issue is coming from the vrts_aliases.py cron job. Something has changed in how gmail is responding to emails here and claiming they're valid. I'm going to try see if we can change that to ignore the gmail check

In T380009#10325989, @Krd wrote:

Can't we just check what was changed yesterday, and undo that?

My postfix knowledge is not really good but what I mean is this order:

transport_maps = regexp:/etc/postfix/transport-wiki-verp-bounce-handler, hash:/etc/postfix/transport-donate, hash:/etc/postfix/transport-phabricator, hash:/etc/postfix/transport-gmail, hash:/var/lib/postfix/transport-vrt, hash:/etc/postfix/transport-recipient-discards

(in main.cf of mx-in)

In T380009#10325964, @Ladsgroup wrote:

As a fast fix, we can put vrt transport rule before gmail rule to make sure it gets checked first.

Can't we just check what was changed yesterday, and undo that?

As a fast fix, we can put vrt transport rule before gmail rule to make sure it gets checked first.

Asking ITS if anything changed on their side recently.

Nothing in recently merged patches of puppet stands out neither anything in private repo. I dig a bit more.

That actually make VRTS fubar. Unbreak now please.

It seems like it's entire @wikimedia.org that is refusing to route to VRTS. My test email to oversight-ko-wp@wikimedia.org also bounced with P71054 (again, acl-otrs-admin + WMF-NDA).

I can definitely say it did not work that way yesterday: there was an incoming info-ko@wikimedia.org ticket at 2024-11-14T02:04:08Z.

for some reason the alias generator script thinks the alias is handled by google and does not route it to VRTS:

Nov 15 09:03:27 mx-in1001 vrts_aliases[4167622]: ERROR:/usr/local/bin/vrts_aliases:Skipping, email is handled by gsuite: info-ko@wikimedia.org

Additionally, it appears to be routed via Google., which perhaps has never been correct.

(Adding SRE and infra-foundations based on tasks at Mail.)

Change #1088362 merged by Dzahn:

[operations/puppet@production] hieradata: delete vrts2001.yaml, host was decom'ed

https://gerrit.wikimedia.org/r/1088362

Change #1088362 had a related patch set uploaded (by Dzahn; author: Dzahn):

[operations/puppet@production] hieradata: delete vrts2001.yaml, host was decom'ed

https://gerrit.wikimedia.org/r/1088362

Decom'd vrts1001 and vrts2001.

cookbooks.sre.hosts.decommission executed by aokoth@cumin1002 for hosts: vrts1001.eqiad.wmnet

• vrts1001.eqiad.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Started forced sync of VMs in Ganeti cluster eqiad to Netbox

Change #1075622 merged by AOkoth:

[operations/puppet@production] site: remove vrts1001 & vrts2001

https://gerrit.wikimedia.org/r/1075622

Icinga downtime and Alertmanager silence (ID=028d754a-b579-4400-abbf-b65ac3fcba3a) set by aokoth@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: Decom

vrts1001.eqiad.wmnet

cookbooks.sre.hosts.decommission executed by aokoth@cumin1002 for hosts: vrts2001.codfw.wmnet

• vrts2001.codfw.wmnet (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster codfw to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Started forced sync of VMs in Ganeti cluster codfw to Netbox

Change #1075622 had a related patch set uploaded (by AOkoth; author: AOkoth):

[operations/puppet@production] site: remove vrts1001 & vrts2001

https://gerrit.wikimedia.org/r/1075622

Icinga downtime and Alertmanager silence (ID=05846004-0f7e-4c57-8098-f62e500d57d8) set by aokoth@cumin1002 for 1 day, 0:00:00 on 1 host(s) and their services with reason: Decom

vrts2001.codfw.wmnet