Incident KPIs

Purpose: This file defines the Key Performance Indicators used to measure the health and effectiveness of the ML incident response programme. These metrics are the source of truth for the monthly leadership report generated by scripts/generate_metrics_report.py and the Grafana dashboard panels in dashboards/dashboard_spec.md.

Tier 1 — SLA Metrics (reviewed weekly)

These metrics have defined SLA targets. Breaches trigger automatic escalation.

Mean Time to Detect (MTTD)

• Definition: Average time from the moment an incident begins (first anomalous signal) to the moment it is formally opened in the tracker.
• Formula: mean(incident.detected_at - incident.started_at)
• Prometheus metric: ml_incident_time_to_detect_minutes
• Target: ≤ 10 minutes for SEV-1, ≤ 30 minutes for SEV-2
• Alert: Fires when the 30-day rolling average exceeds target by 20 %.

Mean Time to Mitigate (MTTM)

• Definition: Average time from incident open to first confirmed mitigation (blast radius contained; user impact stopped or reduced).
• Formula: mean(incident.mitigated_at - incident.created_at)
• Prometheus metric: ml_incident_time_to_mitigate_minutes
• Target: ≤ 30 minutes for SEV-1, ≤ 90 minutes for SEV-2
• Alert: Fires when any single SEV-1 incident exceeds 45 minutes.

Mean Time to Resolve (MTTR)

• Definition: Average time from incident open to full resolution (system restored, monitoring confirms normal, incident closed).
• Formula: mean(incident.resolved_at - incident.created_at)
• Prometheus metric: ml_incident_time_to_resolve_minutes
• Target: ≤ 4 hours for SEV-1, ≤ 24 hours for SEV-2

Tier 2 — Volume and Trend Metrics (reviewed monthly)

These metrics track incident volume and severity distribution over time.

Incident Rate by Severity

• Definition: Number of incidents opened per month, broken down by severity.
• Prometheus metric: ml_incident_created_total{severity, category}
• Target: SEV-1 incidents ≤ 2 per month; month-over-month trend flat or decreasing.

SEV-1 Recurrence Rate

• Definition: Percentage of SEV-1 incidents in a month that share a root cause with a prior SEV-1 in the preceding 90 days.
• Calculation: Manual review at monthly incident retro.
• Target: ≤ 10 % recurrence. Any recurring SEV-1 triggers a systemic review.

Postmortem Completion Rate

• Definition: Percentage of SEV-1 and SEV-2 incidents that have a completed postmortem within 48 hours of resolution.
• Target: 100 % completion rate. Any gap is flagged at the weekly review.

Action Item Closure Rate

• Definition: Percentage of postmortem action items closed within their agreed due date.
• Target: ≥ 80 % on-time closure. Items more than 2 weeks overdue are escalated to the team lead.

Tier 3 — Model and Data Health Metrics (reviewed weekly)

These metrics are leading indicators for ML-specific incidents.

Feature Null Rate

• Definition: Percentage of null values in each feature column per batch.
• Prometheus metric: ml_feature_null_rate{feature, pipeline}
• Alert threshold: > 5 % null rate for any required feature.
• Runbook: runbooks/data_quality_incident.md

Population Stability Index (PSI)

• Definition: Measures distribution shift between the current scoring population and the training baseline for each feature.
• Prometheus metric: ml_feature_psi{feature, model}
• Thresholds:
  • PSI < 0.10: No action required.
  • PSI 0.10–0.20: Monitor closely, schedule investigation.
  • PSI > 0.20: Open SEV-2 incident, suppress model if necessary.
• Runbook: runbooks/model_degradation.md

Token Budget Utilisation (LLM workloads)

• Definition: Percentage of the daily token budget consumed, aggregated across all LLM API calls.
• Prometheus metric: ml_llm_token_budget_utilisation_pct
• Alert threshold: > 80 % daily budget consumed by 18:00 local time.
• Runbook: runbooks/llm_cost_spike.md

Metric Ownership