The prometheus-haproxy-exporter exports avg response time per 1024 successful connections, which is not very helpful. We need to export more meaningful metrics using haproxy's logs.
Description
Description
Details
Details
Customize query in gerrit
| Status | Subtype | Assigned | Task | |
|---|---|---|---|---|
| · · · | ||||
| Resolved | • Gilles | T187765 Replace the Nginx fronting Thumbor with Haproxy | ||
| Resolved | jijiki | T220499 Export useful metrics from haproxy logs for Thumbor | ||
| · · · |
Event Timeline
Comment Actions
For reference: https://medium.com/@tom.fawcett/extracting-useful-duration-metrics-from-haproxy-prometheus-fluentd-2be9832ff702
We can do the same with mtail.
Comment Actions
Change 502967 had a related patch set uploaded (by Gilles; owner: Gilles):
[mediawiki/vagrant@master] Expose haproxy total request time via mtail
Comment Actions
Change 502967 merged by jenkins-bot:
[mediawiki/vagrant@master] Expose haproxy total request time via mtail
Comment Actions
Change 502972 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Expose haproxy total request time via mtail
Comment Actions
Change 502972 merged by Effie Mouzeli:
[operations/puppet@production] haproxy: improve metrics (via mtail) and logging
Comment Actions
Change 504284 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] thumbor: enable haproxy mtail metrics
Comment Actions
Change 504323 had a related patch set uploaded (by Gilles; owner: Gilles):
[operations/puppet@production] Add tests for haproxy mtail program
Comment Actions
Change 504284 merged by Effie Mouzeli:
[operations/puppet@production] thumbor: enable haproxy mtail metrics
Comment Actions
Change 504323 merged by Effie Mouzeli:
[operations/puppet@production] Add tests for haproxy mtail program
Comment Actions
Change 504335 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] thumbor: Fix mtail group and log path
Comment Actions
Change 504335 merged by Effie Mouzeli:
[operations/puppet@production] thumbor: Fix mtail group and log path
Comment Actions
I'm not seeing the metrics show up in the "eqiad/prometheus" ops datasource in Grafana. I'm not sure how prometheus is supposed to be configured to collect the data from the thumbor hosts, though.
Comment Actions
Change 504924 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] prometheus: Fix haproxy mtail stats for thumbor
Comment Actions
Change 504924 merged by Effie Mouzeli:
[operations/puppet@production] prometheus: Fix haproxy mtail stats for thumbor
Comment Actions
Change 504978 had a related patch set uploaded (by Effie Mouzeli; owner: Effie Mouzeli):
[operations/puppet@production] thumbor: Inlcude mtail in ferm configuration
Comment Actions
Change 504978 merged by Effie Mouzeli:
[operations/puppet@production] thumbor: Inlcude mtail in ferm configuration
Comment Actions
@Gilles This is fixed now, I will though revert back to nginx for the weekend. We do have data we can work with from today.
Comment Actions
I've added the relevant panels to your dashboard, mirroring the data we were tracking for nginx: