The good news: this works, we now see some hap: prefix rules appearing in x-requestctl.

Now deployed on all cp hosts in codfw.

I'm reopening this to track the rollout of this feature beyond cp4044.

@ihurbain just deployed the crashpad flag flip patch and (at least for now) Proton looks happier.

+1 to fingerprint as the key of keys

And now some notes pseudo-inlined with the task description.

In T379901#10322298, @JMeybohm wrote:

I would like to see some more details on how this compares to "proper" readiness/liveness probes and in which cases we would use this instead of the builtin probes. I do understand that thumbor might be a bit of a special case here but I fear a tool like this could become a silver bullet for everything that is not working properly and needs kicking from time to time.

If you look at /etc/ssh/userkeys/root.d/cumin on a prod host, it contains the restrict option, which disallows PTY.

fixed in 1.63.0 🎉

Given the shape of the error message and stack trace, I suspect that there's some new code path in FCM or its dependencies for which only setting the httpAgent param isn't sufficient, and we also need to set an environment variable.

@bvibber @aude @Jdlrobson @CCiufo-WMF @Seddon FYI

In T374683#10295634, @daniel wrote:

That would be nice, but it wouldn't have caught the problem we just encoutnered either - I wouldn't have thought to check for redirect=false. Known good cases are easy to cover, unknown bad cases are easy to miss...

Also, we may encounter problems that are more subtle - etag handling or cache control headers or CORS or whatnot. I checked all the stuff that I could think of a while ago, see https://docs.google.com/spreadsheets/d/10FaxUcD6y4Xjss21HfXUwVsH98RCWO7Bs9hhZuDTfFg/edit?pli=1&gid=0#gid=0. But as we just found out, I didn't think of all the relevgant things to check...

LGTM! please use groups C and D if possible, that would give full diversity across ganeti groups

The chart itself looks good, thanks! Also ran it on minikube locally.

I'm pretty confident this is the same as T348730, and I think it would be okay to return ganeti1025 to service and close this task as a dup

I think choose generic-application and add on istio and service-mesh to get started.

In T348763#10262531, @BTullis wrote:

• add an authenticating reverse proxy using envoy or some other kind of service.

At first glance, the envoy based solution looks pretty neat and tidy, given that we already have envoy installed in every pod.

Yep!

In T376762#10246832, @akosiaris wrote:

21.75.192.10.in-addr.arpa. 5 IN PTR 10-192-75-21.eventstreams-production-tls-service.eventstreams.svc.cluster.local.
This is actually "eventstreams-production" on the staging-codfw cluster. Impossible to tell from this DNS response.

This is badly named.

Jesse, not sure how much things have changed since T215183: Redundant bootloaders for software RAID but there's at least a partman recipe in there that used to work.

Should we attempt this on a cluster or two? One of the stagings, and then perhaps aux?

works on beta cluster as well:

@pmiazga @mszabo will either one of you have some time soon to at least do #3 above? That's the one I know the least about.

In T372081#10236490, @JMeybohm wrote:

What are the expectations regarding traffic here? Could this use Ingress instead of LVS?

In T340552#10221676, @pmiazga wrote:

@CDanis @Krinkle - what do you think would be the best areas to instrument? IMHO we should wrap entire request with a root span and instrument Database calls and the HTTPFactory. We could instrument the Parser too. For the first run we should keep it small.

@Ahoelzl why was this moved to "Radar (External Teams)" column? Per @BTullis's post, I think this was awaiting DE approval before DE would work on it...?

In practice the very basic alerting from systemd unit failures has been enough for every statograph issue so far.

Turns out the object counts are already in Prometheus. Here's a quick plot on a dashboard: https://grafana.wikimedia.org/goto/u3dyc3kHg?orgId=1

In T376762#10214038, @akosiaris wrote:

As for the rewrite part, we are currently setting cluster.local via hiera (it is also the default). There is a discussion we can have here about whether we want that setting to be the same across all clusters or whether it makes sense to have it per cluster. We probably should have had this conversation a bit earlier before we started using it more extensively, but it's probably not too late. Does it make sense to make this unique and per cluster? This would mean some deployments will need overrides per cluster in helmfile.d, but that's probably not too bad?

In T376267#10214610, @Ladsgroup wrote:

@CDanis, I renamed your account, can you try doing password reset with "CDanis (WMF)" again?

In T376762#10213193, @JMeybohm wrote:

I'm not sure though what implications the rewrite might have for cluster internal clients (where .cluster.local is perfectly valid and recommended to use). Can we make it so that the rewrite only happens for external clients?

Works in prod now:

OK, one weird issue I've found which is confounding but not fatal: the NodePort isn't working on v6.

In T376291#10196463, @Volans wrote:

Is there plan to try to get away from the very long hardcoded lists in hiera?

A tag or custom field or some other machine-parsable data that indicated the name of the k8s cluster associated with each prefix would be suuuuper helpful for programmatically generating the delegations for T344171.

Janis agrees with me that this use case seems like a great excuse to experiment with externalIPs.