Action items that came out of the investigation and documentation for past Wikimedia production incidents. These action items reduce risk, shorten/reduce impact, or help prevent incidents in the future.
See also:
I think that we can do this, but maybe we should try to avoid changing the production Superset instance, if we can avoid it.
As an alternative, which I might actually prefer, all the process would remain server-side if we can grant access to the superset api via an api token or something similar to the web UI. I'll investigate if that's possible.
The way we could do this is something as follows:
With the new requestctl web UI I think it would be very useful if the current requestctl generator ( https://superset.wikimedia.org/requestctl-generator?q= ) would be adapted to work with the new web UI or (even better) ditched and get the same functionality embedded into requestctl web UI directly just providing the URL of a superset dashboard with filters.
Change #1087615 had a related patch set uploaded (by CDanis; author: CDanis):
[operations/puppet@production] haproxy: bwlim-by-path: also roll out to eqiad
Here we go again in 2024, when you can submit a change that redeclares an existing class with PHP just being like "oh, you're declaring the same class twice, but surely this isn't serious enough for me to stop you, please go ahead" and CI not batting an eye if you merge it. I think the CI solution is fine, but even better would be catching all warnings from the autoloader and turning them into fatals. As I noted in T378774#10283126, doing it unconditionally may be bad for performance ({{cn}}, though); but maybe we could do it for tests only.
Any SRE can feel free to edit the ticket description if I missed something or to clarify. This was just a follow-up trying to remember from my personal meeting notes.
Change #1078989 abandoned by RLazarus:
[operations/puppet@production] do not submit: T376795 cleanups
Reason:
Incident is over, no longer needed
Change #1079314 merged by RLazarus:
[operations/puppet@production] deployment_server: Read Helm secrets in `mwscript-cleanup`
We have migrated postfix
Alert rules have been deployed last week. The limit removal has been rolled out today.
Change #1079459 merged by JMeybohm:
[operations/deployment-charts@master] Remove memory limits from calico components in wikikube clusters
Adding 6k configmaps does not really do anything to calico, cert-manager, helm-state metrics. It might have an impact on the kubelet when the configmaps are actually used, though. I've not tested that but given the fact that this would spread across all kubelets I don't think it's an issue.
Change #1079457 merged by jenkins-bot:
[operations/alerts@master] Add CalicoHighMemoryUsage alert
I've confirmed in the staging cluster that creating 1k network-policies (even if they don't apply to anything) bumps the avg calico-node memory usage from ~130 to ~390MB. CPU usage also increases quite significantly.
Change #1079459 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/deployment-charts@master] Remove memory limits from calico components in wikikube clusters
Change #1079457 had a related patch set uploaded (by JMeybohm; author: JMeybohm):
[operations/alerts@master] Add CalicoHighMemoryUsage alert
Change #1079445 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):
[operations/deployment-charts@master] mw-script: deduplicate resources
Change #1079444 had a related patch set uploaded (by Giuseppe Lavagetto; author: Giuseppe Lavagetto):
[operations/deployment-charts@master] mediawiki: deduplicate network policies and configmaps
We 've already discussed this in a 1on1 and just for transparency's sake, this finds me in agreement. The outage we had this week carried a signal that we don't want to lose, that is that memory usage exploded over the course of a few hours, which in itself is a signal that something else was amiss (which is what T376795: mwscript-k8s creates too many resources is about). At the same time, an outage is the worst possible messenger. So finding some other way to keep the signal, like the alert pointed out above SGTM.
Change #1079314 restored by RLazarus:
[operations/puppet@production] deployment_server: Tweak mwscript-cleanup `helm list` pagination
Change #1079314 abandoned by RLazarus:
[operations/puppet@production] deployment_server: Tweak mwscript-cleanup `helm list` pagination
Reason:
Scott correctly points out I've got this the wrong way around -- the more likely issue is I'm iterating over the list while destroying releases, so the offset is invalid. 🤦 Another, better fix to follow.
Turns out the object counts are already in Prometheus. Here's a quick plot on a dashboard: https://grafana.wikimedia.org/goto/u3dyc3kHg?orgId=1
Change #1079314 had a related patch set uploaded (by RLazarus; author: RLazarus):
[operations/puppet@production] deployment_server: Tweak mwscript-cleanup `helm list` pagination
A mwscript-k8s flag to log to SAL is on my to-do list -- I hadn't gotten around to filing a task, thanks.
Change #1078988 merged by RLazarus:
[operations/puppet@production] deployment_server: Add `helm list` pagination to mwscript-cleanup
Change #1078989 had a related patch set uploaded (by RLazarus; author: RLazarus):
[operations/puppet@production] do not submit: T376795 cleanups