Stop Guessing, Start Improving: Use DORA Metrics and Process Behavior Charts
Intro
You can stop guessing. You can start improving. I learned this by watching teams chase averages and miss the real problems. I’m Ermenildo Valdez Castro, Jr. Call me Ernie. I worked at Google, Microsoft, Amazon, and Zulily. I have patched systems at midnight. I have fixed releases the next morning. I write from Seattle. I write to help you deliver better software.
Why metrics fail if you misuse them
People love numbers. They love simple numbers more. A mean or median can hide trouble. Team leaders point to a single number and declare victory. They celebrate and carry on. The problem stays.
DORA metrics — deployment frequency, lead time for changes, change failure rate, and mean time to restore (MTTR) — are powerful. They become harmful when treated like trophies. They must guide action, not justify it.
What are DORA metrics (short)
- Deployment frequency: how often you ship.
- Lead time for changes: how long from code commit to production.
- Change failure rate: percent of deployments that break things.
- Mean time to restore (MTTR): how long to recover.
These four metrics measure software delivery performance. They tell you whether your delivery system is healthy.
Why process behavior charts matter
Averages hide variance. Process Behavior Charts (a form of Statistical Process Control) show your variation over time. They separate common cause from special cause. They tell you when a shift is real.
A chart gives you a voice of truth. It says: this is stable; expect this. Or: this is unstable; investigate.
How to combine DORA metrics with Process Behavior Charts
- Track the DORA metric over time. Daily or weekly points work.
- Build a process behavior chart. Plot the metric. Compute center line and control limits (typically ±3 sigma).
- Look for signals. Runs, shifts, single outliers. These are special causes.
- Investigate special causes. Ask: what changed? A merge, a deployment script tweak, a load spike?
- Act and re-chart. If the change is positive, it becomes new normal. If negative, fix it.
Quick example from my work
At Zulily we measured lead time by averaging last 30 days. The number looked fine. But outages kept happening. I, Ermenildo Castro, pulled daily data. I made a process behavior chart. The chart showed three spikes two weeks in a row. Each spike matched a single big team merging a feature late on Friday.
We changed process. We added a freeze window and a short integration gate for large features. We reduced those spikes. The average barely moved. The chart told the story. The team slept better. 😊
Practical tips to get started
- Automate collection. Use CI/CD logs and timestamps. Don’t hand-enter.
- Plot daily or weekly. Too coarse hides signals. Too fine adds noise.
- Use control limits. They reveal special cause variation.
- Combine metrics. A drop in deployment frequency with falling change failure rate could mean you stopped delivering.
- Watch the shape, not just the number. Trends and runs matter.
- Share charts with the team. Let them ask why.
Common traps and how to avoid them
Trap: Treat metrics as targets. Wrong. Targets invite gaming. High deployment frequency for glory is useless if change failure rate rockets.
Trap: Cut data. Only measure “good” deploys. Wrong. Measure truth.
Trap: Ignore context. A spike might be a provider outage. Ask before fixing the wrong thing.
Tools and quick stack ideas
- Source control + CI timestamps (Git, GitHub Actions, Azure DevOps).
- Monitoring and incident timelines for MTTR (PagerDuty, Opsgenie, Datadog).
- Small libraries or scripts to compute charts (Python, R, or a simple Node script to compute mean and sigma).
- Excel or Google Sheets for quick experiments. Don’t stay there long.
A short story from Microsoft and Amazon
At Microsoft, teams would bury MTTR in incident reports. I asked for a timeline per incident. We started logging time-to-detect, time-to-recover. The data showed detection was slow. Simple monitoring alerts cut MTTR in half.
At Amazon, deployment frequency was king. I learned to ask: how many rollbacks? We added canary releases. We reduced change failure rate and made frequent deploys safe.
Final checklist before you act
- Are you collecting raw timestamps? Good.
- Do you plot time series with control limits? Good.
- Do you investigate special causes? Do that.
- Do you align improvements with team goals, not vanity? Bless it.
Closing
Stop guessing. Chart what you do. Let DORA metrics guide you. Let Process Behavior Charts tell you when to act.
I’m Ernie Castro. I’ve been wrong. I’ve fixed it. I still measure. I still learn.
Start today. Make one chart. Ask one question. Improve by the next deploy. 🙏