I think we're good here!

Hello! A few next-steps to move forward with this request:

membership to ldap group wmf has been provisioned, thanks!

uid=khantstop has been added to ldap group wmf

In T378989#10301395, @herron wrote:

ganeti1028:~# gnt-instance console aux-k8s-worker1004.eqiad.wmnet
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
SHA256:C9d5dCz/suTrJQqv6NtW3q/R241NXTC1GL3JMLfaMY4.
Please contact your system administrator.
Add correct host key in /dev/null to get rid of this message.
Offending DSA key in /var/lib/ganeti/known_hosts:2
  remove with:
  ssh-keygen -f "/var/lib/ganeti/known_hosts" -R "ganeti01.svc.eqiad.wmnet"
RSA host key for ganeti01.svc.eqiad.wmnet has changed and you have requested strict checking.
Host key verification failed.
Failure: command execution error:
Connection to console of instance aux-k8s-worker1004.eqiad.wmnet failed, please check cluster configuration

Looks like this VM got assigned to new ganeti host ganeti1045.eqiad.wmnet which might still be a work in progress T378921?

(Fixed in T377703)

In T359293#10280713, @gerritbot wrote:

Change #1084199 merged by Herron:

[operations/puppet@production] profile::syslog::centralserver: use prometheus cert for blackbox check

https://gerrit.wikimedia.org/r/1084199

Looked into this a bit since the silence expired, the service being probed is up but looks like the related prometheus blackbox exporter exporter is failing to load the configured certificate

Looked into this a bit and from what I can tell the feature for OTEL sampling within Thanos was introduced in a later version (v0.32.0)

https://trace.wikimedia.org is now running 1.62.0.

Open file limit raised to 16384, we should be good for now

as a stopgap I manually increased max files for the grafana-loki unit and the service recovered, will propose something to persist that as well

optimistically resolving now that the template has been updated

https://slo.wikimedia.org/?search=logstash-availability has been onboarded to pyrra as a bool_gauge. It'll take some time for the recording rule data to populate, but I think we're good here!

At the moment I'm experimenting with something like this

Closing as the workaround above solves this well enough

The full queries with empty responses are:

Yes I agree, in fact I don't see results for cpu_throttled in Thanos at all

The Grafana default datasource has been switched again to Thanos. If needed, the patch to revert is https://gerrit.wikimedia.org/r/1069230

In T269333#10056987, @Lucas_Werkmeister_WMDE wrote:

In T269333#10023141, @gerritbot wrote:

Change #1057882 merged by Herron:

[operations/puppet@production] grafana: set thanos as default datasource

https://gerrit.wikimedia.org/r/1057882

FTR, this was just reverted due to the issue @fgiunchedi mentioned above.

In T371520#10057000, @Lucas_Werkmeister_WMDE wrote:

I’ve just saved new versions of wikidata-edits (schema version 34→38), wikidata-query-service-ui (34→38) and wikidata-special-entitydata (16→38 – no, that’s not a typo! it was ancient). So hopefully those should now stay working if the default is changed again.

Comparing with an example after hours incident like incident 4926 VO logs about 2 dozen contact attempts to the timeline when the incident is escalated to the batphone.

The grafana default datasource has been switched to Thanos. If any unexpected issues occur the patch to revert is https://gerrit.wikimedia.org/r/1057882. There's also a sqlite db backup on the grafana filesystem with todays date, just in case.

I've updated all dashboards with null panel datasources as of today. At this point the only panels that remain with null datasource are dashboard rows which are expected as they are the row containers for other panels.

Agree! +1 for option 2 here as well.

Had a closer look into the threshold tunables and I don't actually see a way to change this natively within Pyrra. As-is the "for" duration of SLOMetricAbsent alerts is 6m. Pyrra has options to enable/disable the absent alert, but maybe we can configure something to put this alert in a silence/inhibit waiting room for 30-60m before it alerts (or recovers on its own)

SLOMetricAbsent for dead_letters_hits, varnish_sli_bad, trafficserver_backend_sli_bad and haproxy_sli_bad occurred today, at two different times.

Reviewing thanos-rule logs I'm seeing related discards with err="out of order sample"

It occurred to me that deploying absent metric alerts for the metrics where we're seeing gaps would be a reasonable next step. That'd let us troubleshoot gaps in/closer to their broken state which should help toward better understanding the issue and steps to resolve manually. Plus, it'd of course help us respond faster and shrink the gaps. I'll work on a patch.

Proposal SGTM for the near-term. Thank you for organizing this!

Buster is looking fine with this deb as well. So I've gone ahead and uploaded 1.8.0 to bookworm-wikimedia, bullseye-wikimedia, and buster-wikimedia. Up next is a small canary upgrade before fully rolling out.

Nice, thanks for the pointer! It looks like export CGO_ENABLED=0 does the right thing. At least, with this set the package builds and installs successfully on my bullseye test host.

In T368088#9917836, @MoritzMuehlenhoff wrote:

Given that this is a Go static ELF we can also simply build on bookworm and copy over the deb to bullseye-wikimedia, we're doing this for other exporters as well. buster might be tricky due to it's old libc6, but we can also ignore it, there's less than 150 hosts left and they can simply live the old IPMI monitoring.

Quick update: prometheus-ipmi-exporter-1.8.0 was a straightforward backport for bookworm https://gitlab.wikimedia.org/repos/sre/prometheus-ipmi-exporter/-/jobs/291532

In T352756#9913669, @elukey wrote:

On the 6th in the thanos-rule logs I see a ton of errors while connecting to Prometheus nodes, and around 14 UTC a reload was issued on both titan active nodes.

In T253810#9485013, @fgiunchedi wrote:

ipmi_exporter now has support to collect generic SEL entries and export metrics from those: https://github.com/prometheus-community/ipmi_exporter/pull/179

Sounds like a plan!

In T365265#9902980, @Clement_Goubert wrote:

I think we may want to enable it deployment per deployment so SRE Observability can monitor the load on prometheus. @colewhite or @herron can we coordinate on this?

Looking longer-term I think it'd be generally worthwhile to support more than baseline blackbox checks on the mgmt interfaces and I'm personally open to exploring something like the redfish exporter. But I think this would be a medium to large sized project since AIUI it will involve a decent sized chunk of setup effort on the hardware itself, and offhand I'm also not sure the percentage of hw in the fleet that will support the approach today. I'm also assuming we would run into some hw vendor oddities/bugs along the way.

In T359879#9875881, @elukey wrote:

@herron let's double check, maybe we can drop the secondary rules and keep going with the "regular" ones?

Example of the fix: https://grafana.wikimedia.org/d/slo-Lift_Wing_Revert_Risk_LA/lift-wing-revert-risk-la-slo-s?orgId=1&from=2024-03-01%2000:00:00&to=2024-05-31%2023:59:59

done

Group membership has been provisioned, thanks!

Hi @odimitrijevic @Milimetric @WDoranWMF @Ahoelzl could one of you please approve this request for analytics-privatedata-users? Thanks in advance!

This was completed yesterday (during stashbot outage, this task unfortunately missed the !log)

The patch to provision this access has been merged, and will be fully propagated within the next 30 minutes. I'll transition this to resolved now, please reopen if any followup is needed. Thanks!

The patch to provision this access has been merged and will fully propagate within the next 30 minutes. I'll transition this to resolved now, please re-open if any followup is needed. Thanks!