Architecture

The ML Incident Response platform is a production-grade FastAPI service with SQLite (dev) / PostgreSQL (prod) persistence, Redis-backed JWT revocation, Prometheus metrics, OpenTelemetry distributed tracing, and a hardened CI pipeline. This document describes every layer, how they connect, and the reasoning behind key design decisions.

Last updated: 2026-05-28 (Phase 2 — ADR cross-references, CR-1/coverage corrections)

Component Diagram

Layer Descriptions

API Layer (api/)

api/app.py is the single FastAPI application. It registers:

• A lifespan context manager that runs init_db() on startup and shutdown_otel() on shutdown.
• Three middleware layers: CORS (configured for the ENVIRONMENT), a trace_and_security_headers middleware that injects a UUID4 X-Trace-Id header on every response and binds it to the structlog context, and HTTP security headers (X-Content-Type-Options, X-Frame-Options, HSTS, CSP, etc.).
• Auth routes (/auth/*) for token issuance, refresh, and logout (denylist).
• Incident CRUD routes (/incidents/*).
• Health and readiness probes (/health, /ready) and a Prometheus metrics endpoint (/metrics).

Authentication Layer

JWT access tokens (15-minute TTL) and refresh tokens (7-day TTL) are issued using RS256 in production and HS256 in unit tests (see ADR-004). Passwords are hashed with argon2id via argon2-cffi (OWASP 2024 recommendation). Logout adds the token’s JTI (JWT ID) to a Redis sorted set with a TTL matching the token’s expiry. is_token_revoked() fails closed: if Redis is unreachable, the token is treated as revoked and access is denied. The RedisDenylistUnavailable Prometheus alert fires within 1 minute if Redis goes down.

Data Layer (src/)

IncidentRepository wraps all database access behind an async typed interface (see ADR-001). It uses SQLAlchemy 2.x with AsyncSession so the FastAPI event loop is never blocked by I/O. Schema management is owned by Alembic (CR-1): init_db() performs a connectivity check and reads alembic_version for an ops warning; it does not call Base.metadata.create_all(). Run alembic upgrade head before starting the application. update_status() enforces lifecycle transitions via the domain state machine in src/domain/incident_lifecycle.py (CR-2). All write operations are audit-logged via src/audit.py.

Audit Layer (src/audit.py)

A dedicated typed audit event stream built on structlog. All state-changing API operations emit structured audit events with log_type="audit". Events are schema-validated against observability/audit_log_schema.json. The audit stream is separate from the application log stream to allow independent routing, retention, and SIEM ingestion.

Monitoring Layer (observability/)

drift_check.py provides three functions:

• drift_ratio() — relative mean deviation for scalar features.
• psi_score() — Population Stability Index for binned distributions.
• scan_features() — batch evaluation with Prometheus gauge export.

alert_rules.yml is valid Prometheus 2.x alerting rule syntax (loadable with promtool check rules). It defines six alert groups covering API error rate, latency, model accuracy, feature drift, pipeline SLA, incident volume, LLM cost, and Redis denylist availability.

Observability Layer

otel_setup.py bootstraps the OTel SDK with a BatchSpanProcessor → OTLP gRPC exporter. It no-ops gracefully if the OTel packages are absent or OTEL_SDK_DISABLED=true. logging_config.py configures structlog to emit machine-parseable JSON in production and a human-readable format in development, with automatic exception formatting and caller context.

CI Layer (.github/workflows/secured_ci.yml)

Seven jobs run on every push and PR to main: secret scanning (TruffleHog, SHA-pinned) gates all other jobs; SAST (Bandit hard gate medium/medium + Semgrep hard gate ERROR severity + mypy) and dependency audit (pip-audit) run in parallel after secrets pass; unit tests (≥75% coverage) and integration tests gate on SAST + audit; container scan (Trivy CRITICAL/HIGH + SBOM) gates on integration tests; deploy gate aggregates all results. All actions/* and tool actions are pinned to full commit SHAs. See docs/ci-conventions.md for the full convention spec.

Dependency Management

requirements.txt is the single source of truth for runtime dependencies. pyproject.toml does not declare application dependencies; it is a pure tooling configuration file. This separation avoids dual-source ambiguity and keeps pip-audit scans scoped to the correct requirement set.

Request Lifecycle

A POST /incidents request from an authenticated client:

1. Middleware — CORS check, X-Trace-Id UUID4 generated and bound to structlog context.
2. OTel — FastAPIInstrumentor creates a root span with http.method, http.route, http.status_code.
3. Auth dependency — get_current_user() extracts the Bearer token, verifies the JWT signature (RS256), checks the Redis denylist.
4. Route handler — validates IncidentCreate schema (Pydantic v2), calls IncidentRepository.create().
5. Repository — inserts the row via AsyncSession, returns the committed Incident ORM object. Emits incident.created audit event.
6. Response — serialized to IncidentResponse schema, HTTP 201 returned. X-Trace-Id is visible in the response headers for client-side correlation.
7. Prometheus — http_requests_total and http_request_duration_seconds incremented/observed by the instrumentator.
8. Logs — structlog emits a JSON line with trace_id, user_id, incident_id, severity, and duration_ms.

Architecture Decision Records

Formal ADRs are in docs/adr/. Each ADR documents context, the decision made, alternatives considered, and consequences.

Technology Decisions