Future Audiences is likely to start coming back to this workstream in the coming weeks so the urgency on our end may be increasing.

stalled on calico upgrade https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/462

sstefanova merged https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/466

sstefanova merged https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/4

dcaro opened https://gitlab.wikimedia.org/toolforge-repos/sample-complex-app-frontend/-/merge_requests/1

sstefanova opened https://gitlab.wikimedia.org/repos/cloud/toolforge/components-api/-/merge_requests/4

I think I'm experiencing a similar error. Suddenly started getting this on a scraping job I have:

2024-08-01T02:42:58+00:00 [test-scrape-pd5kq] â Host system is missing dependencies to run browsers. â
2024-08-01T02:42:58+00:00 [test-scrape-pd5kq] â Missing libraries:                                   â
2024-08-01T02:42:58+00:00 [test-scrape-pd5kq] â     libOSSlib.so                                     â
2024-08-01T02:42:58+00:00 [test-scrape-pd5kq] ââââââââââââââââââ

Another option would be to move --container-runtime-endpoint to /etc/default/kubelet for now, which would let us delete the kubeadm-flags.env file

I tried this manually on my kubeadm test cluster (1.28) and it worked fine. However, in Toolsbeta we are getting this error:

project_1317_bot_df3177307bed93c3f34e421e26c86e38 opened https://gitlab.wikimedia.org/repos/cloud/toolforge/toolforge-deploy/-/merge_requests/462

sstefanova merged https://gitlab.wikimedia.org/repos/cloud/toolforge/calico/-/merge_requests/8

dcaro opened https://gitlab.wikimedia.org/repos/cloud/toolforge/api-gateway/-/merge_requests/30

In T370245#10030659, @Slst2020 wrote:

@dcaro not sure where to go from here. It seems we currently don't have an automated way to roll out cluster-wide config changes.

Change #1058560 had a related patch set uploaded (by Slavina Stefanova; author: Slavina Stefanova):

[operations/puppet@production] aptrepo: upgrade k8s components for 1.26

https://gerrit.wikimedia.org/r/1058560

@dcaro not sure where to go from here. It seems we currently don't have an automated way to roll out cluster-wide config changes.

To recap, what needs to happen for this change is:

Closing this for now because it doesn't seem to be happening repeatedly.

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/58

raymond-ndibe opened https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/58

raymond-ndibe merged https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-cli/-/merge_requests/44

Both --container-runtime and --pod-infra-container-image are set in /var/lib/kubelet/kubeadm-flags.env. This file is not managed by puppet, but is used by kubeadm (init and join) to dynamically set kubelet flags without directly modifying the main kubelet systemd service file.