This repository contains a ready-to-use solution for setting up an API-based service on AWS that will calculate the four (well, there are 5 these days, but who's counting?) DORA metrics completely from processing data that GitHub holds on a given repository:
- Change failure rate (CFR)
- Deployment frequency (DF)
- Lead time to change (LTC)
- Mean time to repair (MTTR)
This solution also supports returning the response in a format that works for those sweet shields.io badges.
Please note that it's kind of messy to sometimes calculate these values strictly to their intended use. Read more about the details of these in the Calculations section and how this solution adapts some shortcomings in the available data.
- Creating deployments in GitHub and using GitHub issues with
bugorincidentlabels - Recent Node.js (ideally 18+) installed.
- Amazon Web Services (AWS) account with sufficient permissions so that you can deploy infrastructure. A naive but simple policy would be full rights for CloudWatch, Lambda, API Gateway, and S3.
- Ideally some experience with Serverless Framework as that's what we will use to deploy the service and infrastructure.
To make use of this project, you'll need a GitHub personal access token with repo and issues access and use this in any calls to GitHub.
You need to use one of these options to actually use such a token:
- Let the caller pass in their token.
- Pre-bake a token:
- Passing it in as an option during CI
npx sls deploy --pat YOUR_PAT - Hardcode the value in
serverless.yml - Get the value from AWS Secrets Manager when building/compiling
- Passing it in as an option during CI
See this guide for how to create a personal access token.
The statement at the top—that this is a ready-to-use solution—is therefore technically true since consumers of the API may pass in their own GitHub personal access token (PAT) to make calls, for example to private/internal repositories. However, you'll probably want to set up and bake in a public access PAT to make the requests towards GitHub for any case in which no token is directly passed in.
You can simply pass it in as an option during CI; for example with npx sls deploy --pat YOUR_PAT.
For hardcoding it, you'll set the PAT value in serverless.yml under custom.config.gitHubPersonalAccessToken. The recommended way, however, would be to use AWS Secrets Manager or similar. Both ways are present in serverless.yml but the Secrets Manager solution is commented out. Please refer to the comments there for more information.
- Use GitHub Issues with
bugorincidentlabels- Open and close times are used
- Use GitHub deployments, deploying to one of the named environments
prd,prod,production, orlive- The commit time (prompting the deployment) and start-of-deployment times are used
Clone, fork, or download the repo as you normally would. Run npm install.
npm start: Run application locallynpm test: Test the business/application logic with AVAnpm run build: Package application with Serverless Frameworknpm run deploy: Deploy application to AWS with Serverless Frameworknpm run teardown: Remove stack from AWS
Using npm start you can start using the local endpoint http://localhost:3000/metrics to call the service. See example calls further down.
Quotes from a blog post on Google Cloud.
The period that is taken into account is the one provided in the serverless.yml configuration, under custom.config.maxPeriodInDays. It's set to 30 (days) by default.
How often an organization successfully releases to production.
This is really straight-forward but assumes/requires you to use GitHub "deployments" to express these deployments.
{number of deployments in period} which use an environment named one of the following:
prdprodproductionlive
The amount of time it takes a commit to get into production.
This reflects the time from a commit being made to a deployment being started.
There is no crystal-clear way to get the time of deployment as the updatedAt value of the deployment may express other changes as well – this is why the start of the deployment will have to be the best proxy value I could come up with.
{time of deploy - time committed} as full seconds, using the median of all processed values.
The percentage of deployments causing a failure in production.
This one is hard as doing it orthodoxly would mean mapping each deployment to a specific failure, which is obviously very hard, sometimes not even possible, and is definitely something that would explode the scope far out from just having to do with GitHub. While it's indeed possible to check for strictly failed deployments, I don't find that to be a true indicator of real issues as they can be (given you don't do canary releases etc.) attributed to things happening before the code even reaches actual users.
The solution used here is to accept a somewhat wider perspective by simply dividing the number of closed issues tagged bug or incident with the number of deployments.
{number of issues / number of deployments} * 100 as a rounded percentage.
If there are no deployments in the period, this will be represented as zero percent.
How long it takes an organization to recover from a failure in production.
Same as above, this can be tricky to have an opinion on and getting it working completely inside of GitHub.
The solution is to calculate the median time to close issues marked bug or incident within the period. While it won't only reflect handling production failures, it will at least give a bug resolution time.
{issue close time - issue open time}.
curl {ENDPOINT}/metrics?repo={OWNER}/{REPO}{
"changeFailureRate": "66.67%",
"deploymentFrequency": "0.10/day",
"leadTimeForChange": "00:00:00:20",
"meanTimeToRepair": "00:06:21:15"
}curl {ENDPOINT}/metrics?repo={OWNER}/{REPO}&badge=true{
"schemaVersion": 1,
"label": "DORA metrics",
"message": "CFR: 66.67% | DF: 0.10/day | LTC: 00:00:00:20 | MTTR: 00:06:21:15",
"color": "black",
"labelColor": "blue",
"style": "for-the-badge"
}Either pass it in your request:
curl {ENDPOINT}/metrics?repo={OWNER}/{REPO}&token={YOUR_GH_PAT}Or use a header for this:
Authorization: {YOUR_GH_PAT}
- There doesn't seem to be a (reliable) way to get pagination for GitHub deployments, meaning only the last 100 deployments (maximum) will be used to calculate deployment-related metrics.