Once this patch is completed/merged, we'll have the airflow webserver running. The scheduler is another requirement, so this ticket is dedicated to adding the scheduler to our helm release. See the Airflow: High Availability Strategy doc for more details on this process.
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | ||
|---|---|---|---|---|---|
| Open | None | T362788 Migrate Airflow to the dse-k8s cluster | |||
| Open | None | T364389 Migrate the airflow scheduler components to Kubernetes | |||
| Resolved | brouberol | T368737 Deploy airflow scheduler via helm chart |
Event Timeline
Comment Actions
Change #1070245 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] airflow: fully generate airflow.cfg from helm values
Comment Actions
Change #1070245 merged by Brouberol:
[operations/deployment-charts@master] airflow: fully generate airflow.cfg from helm values
Comment Actions
Change #1070255 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/puppet@production] global_config: define an external-services entry for mx[1-2]001.wikimedia.org
Comment Actions
Change #1070255 merged by Brouberol:
[operations/puppet@production] global_config: define an external-services entry for mx[1-2]001.wikimedia.org
Comment Actions
Change #1070619 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] airflow: deploy the scheduler as a sidecar container
Comment Actions
I've been able to deploy the Airflow scheduler to our airflow-test-k8s instance! Everything looks nominal.
NAME READY STATUS RESTARTS AGE airflow-scheduler-974c47bcf-8lwj5 1/1 Running 0 104s airflow-webserver-b6dc74bdd-htcn5 2/2 Running 0 104s
We can see our (single, for now) DAG: addition, that computes ... 1 + 2.
I've been able to manually trigger it, and it ran successfully.
The scheduler was able to serialize the dag and store it to DB, so that the webserver is able to display its code.
All required components are running (the triggerer is optional, and we don't run it):
Comment Actions
brouberol opened https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/821
test-k8s: start regularly scheduling DAGs now that we run the scheduler
Comment Actions
brouberol merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/821
test-k8s: start regularly scheduling DAGs now that we run the scheduler
Comment Actions
Change #1070619 merged by Brouberol:
[operations/deployment-charts@master] airflow: deploy the scheduler via a separate Deployment
Comment Actions
Change #1071077 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] airflow: fix badly formatted Deployment separation
Comment Actions
Change #1071077 merged by Brouberol:
[operations/deployment-charts@master] airflow: fix badly formatted Deployment separation
Comment Actions
Change #1071153 had a related patch set uploaded (by Brouberol; author: Brouberol):
[operations/deployment-charts@master] airflow: enable visualizing logs of DAG runs in the webserver UI
Comment Actions
Change #1071153 merged by jenkins-bot:
[operations/deployment-charts@master] airflow: enable visualizing logs of DAG runs in the webserver UI