Milestone for Data Platform SRE work
Details
Details
Description
Sep 16 2024
Sep 16 2024
fnegri closed T365424: Upgrade clouddb* hosts to Bookworm, a subtask of T368518: decommission clouddb1021, as Resolved.
brouberol closed T374396: Retry package added needs the types-retry 0.9.9.4 typing stub as Resolved.
All airflow instances have been restarted and are now using the new deb package, with the types-retry dependency made available.
brouberol closed T374396: Retry package added needs the types-retry 0.9.9.4 typing stub, a subtask of T372279: Add Retry package to airflow conda environment, as Resolved.
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073173 merged by Brouberol:
[operations/puppet@production] Install airflow-dags 2.9.3-py3.10-20240916 by default on all instances
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073172 merged by Brouberol:
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-analytics
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073171 merged by Brouberol:
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-wmde
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073170 merged by Brouberol:
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-search
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073169 merged by Brouberol:
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-research
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073168 merged by Brouberol:
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-platform-eng
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073167 merged by Brouberol:
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-analytics-product
brouberol added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
root@apt1002:~# wget https://gitlab.wikimedia.org/api/v4/projects/93/packages/generic/airflow/2.9.3/airflow-2.9.3_amd64.deb --2024-09-16 10:47:06-- https://gitlab.wikimedia.org/api/v4/projects/93/packages/generic/airflow/2.9.3/airflow-2.9.3_amd64.deb Resolving gitlab.wikimedia.org (gitlab.wikimedia.org)... 2620:0:860:1:208:80:153:8, 208.80.153.8 Connecting to gitlab.wikimedia.org (gitlab.wikimedia.org)|2620:0:860:1:208:80:153:8|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 579378076 (553M) [application/octet-stream] Saving to: ‘airflow-2.9.3_amd64.deb’
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073166 merged by Brouberol:
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-analytics-test
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073173 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Install airflow-dags 2.9.3-py3.10-20240916 by default on all instances
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073172 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-analytics
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073171 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-wmde
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073170 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-search
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073169 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-research
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073168 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-platform-eng
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073167 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-analytics-product
gerritbot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Change #1073166 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] Upgrade the airflow-dags deb in airflow-analytics-test
brouberol added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Everything seems to be working fine
brouberol added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
brouberol@an-test-client1002:~$ wget https://gitlab.wikimedia.org/api/v4/projects/93/packages/generic/airflow/2.9.3/airflow-2.9.3_amd64.deb --2024-09-16 08:48:53-- https://gitlab.wikimedia.org/api/v4/projects/93/packages/generic/airflow/2.9.3/airflow-2.9.3_amd64.deb Resolving gitlab.wikimedia.org (gitlab.wikimedia.org)... 2620:0:860:1:208:80:153:8, 208.80.153.8 Connecting to gitlab.wikimedia.org (gitlab.wikimedia.org)|2620:0:860:1:208:80:153:8|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 579378076 (553M) [application/octet-stream] Saving to: ‘airflow-2.9.3_amd64.deb’
brouberol added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
A new airflow-dags including the library has been published: https://gitlab.wikimedia.org/api/v4/projects/93/packages/generic/airflow/2.9.3/airflow-2.9.3_amd64.deb
CodeReviewBot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
CodeReviewBot added a project to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub: Patch-For-Review.
Sep 10 2024
Sep 10 2024
CodeReviewBot added a comment to T374396: Retry package added needs the types-retry 0.9.9.4 typing stub.
Sep 9 2024
Sep 9 2024
Aug 16 2024
Aug 16 2024
Gehel edited parent tasks for T368757: Create a git-sync container image to be used with airflow, added: T368033: Design a suitable DAG deployment method; removed: T364387: Adapt Airflow auth and DAG deployment method.
@CDanis : I'm closing this as it seems completed from our side. Please re-open and ping me on Slack if you need more from us.
Aug 15 2024
Aug 15 2024
Stashbot added a comment to T370754: Import WDQS subgraphs to production nodes.
Mentioned in SAL (#wikimedia-operations) [2024-08-15T22:42:57Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T370754, transfer fresh wdqs-main journal to codfw host) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs2021.codfw.wmnet w/ force delete existing files, repooling neither afterwards
bking renamed T371061: Update CirrusSearch dashboards to use new metrics/refresh dashboards from Update CirrusSearch dashboards to use new metrics to Update CirrusSearch dashboards to use new metrics/refresh dashboards.
bking updated the task description for T372442: Investigate why 70% of WDQS EQIAD hosts became unresponsive within a few minutes of each other .
bking renamed T372442: Investigate why 70% of WDQS EQIAD hosts became unresponsive within a few minutes of each other from Determine if WDQS was affected by wikipedia editing outage/consider protections in similiar future scenarios to Investigate why 70% of WDQS EQIAD hosts became unresponsive within a few minutes of each other .
Stashbot added a comment to T370754: Import WDQS subgraphs to production nodes.
Mentioned in SAL (#wikimedia-operations) [2024-08-15T21:54:27Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T370754, transfer fresh wdqs-main journal to codfw host) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs2021.codfw.wmnet w/ force delete existing files, repooling neither afterwards
Stashbot added a comment to T370754: Import WDQS subgraphs to production nodes.
Mentioned in SAL (#wikimedia-operations) [2024-08-15T21:53:55Z] <ryankemper@cumin2002> END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T370754, transfer fresh wdqs-main journal to codfw host) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs2021.codfw.wmnet, repooling neither afterwards
Stashbot added a comment to T370754: Import WDQS subgraphs to production nodes.
Mentioned in SAL (#wikimedia-operations) [2024-08-15T21:53:15Z] <ryankemper@cumin2002> START - Cookbook sre.wdqs.data-transfer (T370754, transfer fresh wdqs-main journal to codfw host) xfer wikidata_main from wdqs1021.eqiad.wmnet -> wdqs2021.codfw.wmnet, repooling neither afterwards
gerritbot added a comment to T364077: Adapt the wdqs data-transfer cookbook to operate with federated subgraphs.
Change #1063067 merged by Ryan Kemper:
[operations/puppet@production] wdqs graph-split: data xfer needs python3-snappy
gerritbot added a comment to T364077: Adapt the wdqs data-transfer cookbook to operate with federated subgraphs.
Change #1063067 had a related patch set uploaded (by Ryan Kemper; author: Ryan Kemper):
[operations/puppet@production] wdqs graph-split: data xfer needs python3-snappy
bking added a comment to T372442: Investigate why 70% of WDQS EQIAD hosts became unresponsive within a few minutes of each other .
As Wikipedia suffered another incident yesterday at around the same time (2100 UTC) , but WDQS did not fall over, it seems these events are probably unrelated. We should still investigate what happened with WDQS, but it doesn't look like the larger incident was related. I'll update this task to reflect this.
Maintenance_bot removed a project from T371208: Create remaining airflow DNS records to be used for access and oidc: Patch-For-Review.
gerritbot added a comment to T371208: Create remaining airflow DNS records to be used for access and oidc.
Change #1062048 merged by Stevemunene:
[operations/dns@master] dns: provision airflow-test-k8s temp domain
gerritbot added a comment to T368760: Configure airflow webserver under Kubernetes to use OIDC authentication.
Change #1062048 merged by Stevemunene:
[operations/dns@master] dns: provision airflow-test-k8s temp domain
Ifrahkhanyaree_WMDE added a comment to T371894: Requesting Kerberos access for ifrahkhanyaree.
That got me a step further! New error after I add my passphrase is