Remediation Log

Remediation Log — ML Incident Response API

Single source of truth for all remediation and roadmap work on the ML Incident Response Playbook. Status updated after every remediation cycle. Never remove completed items — mark VALIDATED.


Status Legend

StatusMeaning
BACKLOGIdentified, not yet started
IN PROGRESSActively being worked
BLOCKEDWaiting on dependency
FIXEDCode/config change committed to main
VALIDATEDFix verified by test / CI pass
REVERTEDCommitted then reverted; back to BACKLOG with note
DEFERREDIntentionally postponed with documented rationale

Priority Tiers

TierCriteria
CRITICALSecurity vulnerability, broken runtime, CI hard failure
HIGHCoverage regression, CI gate mismatch, missing safety net
MEDIUMArchitectural debt, observability gap, DX friction
LOWPolish, documentation, portfolio signal

Active Tracker

🔴 CRITICAL / HIGH — Fix First

IDPhaseCategoryIssueSevStatusBlocking DepsFiles AffectedValidation
R-P1Cycle 2CI/CDIntegration coverage gate 53% (CI-66b); recovery to ≥65% deferred as CI-67HIGHBACKLOGCI-67 fixture worksecured_ci.ymlGate restored ≥65%; CI green
R-P12Cycle 2TestingCI-67 open: Redis / lifespan / auth paths unreachable in integration fixturesHIGHBACKLOGCI-67tests/integration/Integration coverage ≥65%; CI-67 closed
R-P23Phase 12ArchitectureRefactor src/incident_tracker.py → thin facade over domain/services/repositoriesHIGHBACKLOGR-P22src/domain/, src/services/, src/repositories/All existing tests pass; mypy clean
R-P57Phase 18Code HygieneStrip all inline micro-changelogs from source files — history belongs in git and CHANGELOG.md onlyHIGHBACKLOGAll src/, api/, tests/, config filesgrep -r "# .*20[0-9][0-9]-" src/ api/ returns zero hits; CI green

🟠 MEDIUM — Do Next

IDPhaseCategoryIssueSevStatusBlocking DepsFiles AffectedValidation
R-P19Phase 13Architecturesrc/incident_tracker.py module-level _engine singleton (Phase 13 DI migration)MEDIUMDEFERREDR-P23src/incident_tracker.pyEngine constructed only inside lifespan context
R-P24Phase 12ArchitectureCollapse deps to single source of truth: pyproject.toml + pip-compileMEDIUMBACKLOGR-P23pyproject.toml, requirements.txtpip-compile round-trips cleanly; lockfile-check CI green
R-P25Phase 12ArchitectureAdd minimum credible content to infrastructure/ — Terraform stub + READMEMEDIUMBACKLOGinfrastructure/main.tf, infrastructure/README.mdFiles present; terraform validate passes
R-P26Phase 12ArchitectureAdd minimum credible content to dbt/ — README + one model stubMEDIUMBACKLOGdbt/README.md, dbt/models/incidents.sqlFiles present; renders in docs
R-P27Phase 12ArchitectureAdd minimum credible content to orchestration/ — README explaining DAG patternMEDIUMBACKLOGorchestration/README.mdFile present; explains Prefect/Airflow integration
R-P28Phase 12ArchitectureUpdate Architecture Mermaid diagram in README to reflect real code pathMEDIUMBACKLOGR-P23README.mdDiagram matches: FastAPI → Auth → Services → Domain → Postgres/Redis
R-P33Phase 13CI/CDFix README CI/CD section coverage discrepancy (says ≥68%, gate is 75%)MEDIUMBACKLOGREADME.mdREADME states correct gate value
R-P34Phase 14RunbooksExpand all runbooks to operational template formatMEDIUMBACKLOGrunbooks/*.mdAll runbooks have metadata table, query examples, decision tree, escalation
R-P35Phase 14RunbooksAdd runbooks/model_rollback.mdMEDIUMBACKLOGR-P34runbooks/model_rollback.mdFile present; meets operational template standard
R-P36Phase 14RunbooksAdd runbooks/feature_store_corruption.mdMEDIUMBACKLOGR-P34runbooks/feature_store_corruption.mdFile present; meets operational template standard
R-P37Phase 14RunbooksAdd runbooks/runbook_test_log.md — game-day exercise evidenceMEDIUMBACKLOGR-P34runbooks/runbook_test_log.mdFile present; at least one exercise entry documented
R-P38Phase 14ObservabilityAdd configs/slos.yml — numeric SLO definitionsMEDIUMBACKLOGconfigs/slos.ymlFile present; values match Grafana dashboard thresholds
R-P39Phase 15ObservabilityAdd api/metrics.py — Prometheus endpoint with Counter, Histogram, GaugeMEDIUMBACKLOGapi/metrics.py/metrics endpoint responds; curl shows metric names
R-P40Phase 15ObservabilityRegister metrics router in api/main.pyMEDIUMBACKLOGR-P39api/main.pyGET /metrics returns 200 with Prometheus text format
R-P41Phase 15ObservabilityInstrument create_incident path with metric labelsMEDIUMBACKLOGR-P39, R-P40api/routers/incidents.pyMetrics visible in Prometheus scrape after POST
R-P42Phase 15ObservabilityBind runbook threshold values to real Prometheus query expressions in configs/slos.ymlMEDIUMBACKLOGR-P38, R-P39configs/slos.ymlSLO file references real metric names from api/metrics.py
R-P43Phase 15ObservabilityUpdate Grafana dashboard JSON to use real metric namesMEDIUMBACKLOGR-P39, R-P42dashboards/ml_operations_overview.jsonDashboard panels show live data in local Compose stack
R-P44Phase 16MLOpsConfirm HF Space org/slug — README shows zrlo/ml-incident-api, verify correctMEDIUMBACKLOGREADME.md, deploy-hf.ymlHF Space URL resolves; workflow targets correct slug
R-P45Phase 16MLOpsVerify Dockerfile port — HF app_port: 8080; confirm FastAPI binds 0.0.0.0:8080MEDIUMBACKLOGDockerfileContainer starts and responds on 8080
R-P46Phase 16MLOpsProvision external Postgres (Neon free tier) — connect string → HF SecretMEDIUMBACKLOGHF Space SecretsGET /ready returns 200 on live HF Space
R-P47Phase 16MLOpsProvision external Redis (Upstash free tier) — connect string → HF SecretMEDIUMBACKLOGHF Space SecretsRate limiting functional on live HF Space
R-P48Phase 16MLOpsAdd scripts/seed_demo_user.py — seeds read-only demo user on first bootMEDIUMBACKLOGscripts/seed_demo_user.pyScript idempotent; demo user exists after run
R-P49Phase 16MLOpsAdd make deploy-hf + make hf-status Makefile targetsMEDIUMBACKLOGR-P44, R-P45Makefilemake deploy-hf pushes to HF Space remote
R-P50Phase 16MLOpsSmoke-test live HF endpoint: GET /health, POST /auth/token, GET /incidentsMEDIUMBACKLOGR-P46, R-P47, R-P48All three requests return expected responses on live Space

🟡 LOW — Polish & Portfolio

IDPhaseCategoryIssueSevStatusBlocking DepsFiles AffectedValidation
R-P51Phase 17PortfolioAdd “What Senior Reviewers Will Find” section to READMELOWBACKLOGR-P28, R-P33README.mdSection present; copy is accurate and current
R-P52Phase 17PortfolioAdd “Known Limitations” callout section to READMELOWBACKLOGREADME.mdSection present; no overselling
R-P53Phase 17PortfolioUpdate Roadmap section to strategic format: Q3 2026 / Q4 2026 / AspirationalLOWBACKLOGR-P50README.mdRoadmap reflects actual planned phases
R-P54Phase 17PortfolioVerify all badge URLs resolve and are accurateLOWBACKLOGR-P20README.mdAll badges return 200; values match CI state
R-P55Phase 17PortfolioVerify all internal doc links are not brokenLOWBACKLOGR-P34–R-P37README.md, docs/mkdocs build --strict passes with 0 broken links
R-P56Phase 17PortfolioFinal README read-through — remove stale Fly.io references; verify zrl.dev linkLOWBACKLOGR-P51–R-P55README.mdZero Fly.io references; zrl.dev links resolve

Completed Archive

Cycle 3 (2026-05-29)

IDIssueCommitResolution
R-P22Characterization tests for src/incident_tracker.py1db8d11tests/unit/test_incident_tracker_char.py added; full CRUD, keyset pagination, state machine, init_db, get_session coverage
R-P4Semgrep fails for forks/Dependabot with empty token1db8d11if: env.SEMGREP_APP_TOKEN != '' condition added; hard gate retained for owned branches
R-P7CONTRIBUTING.md absent1db8d11Full onboarding guide added
R-P13docker-compose.yml missing/unverified1db8d11Verified and hardened: postgres:16-alpine + redis:7-alpine with health-checks
R-P20README badges absent1db8d11Python version and security badges added; CI/codecov/Codacy confirmed present
R-P16/healthz//readyz K8s probe rename1db8d11 → REVERTED 098fe0bReverted 2026-05-29 — no Kubernetes in project; broke 5 unit tests expecting /health and /ready. Routes restored to originals.

Cycle 2 (2026-05-29)

IDIssueCommitResolution
R-P11SlowAPI get_remote_address stored raw IP as rate-limiter Redis key — HIGH-01 final PII vector660005862Replaced with _rate_limit_key(): SHA-256(best-available-identifier)[:16]; raw IPs no longer enter limiter state
R-P21No regression test for HIGH-01 privacy protections across middleware + rate limiter660005862Added tests/unit/test_middleware_pii.py — 5 tests

Cycle 1 (2026-05-29)

IDIssueCommitResolution
R-P2Duplicate lint: target in Makefile — mypy silently skipped on pipelines/0890c89Single deduplicated target; pipelines/ added to mypy + ruff scope
R-P3test-int gate 65% local vs 53% CI0890c89Aligned to 53%; CI-67 recovery path documented
R-P5Pre-commit mypy --ignore-missing-imports diverges from CI strict configb028d6cRemoved flag; added stub deps to additional_dependencies
R-P6MASTER_ACTION_TRACKER.md reference in CHANGELOG171759dFile merged into this log (2026-05-29); local copy deleted
R-P8CODEOWNERS minimalaf04cc5Hardened with explicit security-sensitive and dependency manifest paths
R-P9RequestTimeoutMiddleware logs raw client IPc7ea849_pseudo_ip() applied
R-P10MaxBodySizeMiddleware logs raw client IP in 2 branchesc7ea849_pseudo_ip() applied to both branches
R-P29pytest-xdist in requirements-dev.txtpre-existingConfirmed already present; -n auto wired in CI
R-P30test_model_registry_thread_safety.pypre-existingConfirmed already present in CI test list
R-P31Redis denylist concurrency testspre-existingConfirmed test_redis_denylist_concurrency.py already present
R-P32test_incident_service_contract.pypre-existingConfirmed already present in CI test list

Pre-Engagement Archive

Phase 6 — Repo Hygiene + CI Accuracy (2026-05-27)

IDFindingSevStatusFiles Changed
R-02Orphaned .github/release-placeholder-v110.txt removedLOW✅ CLOSED.github/release-placeholder-v110.txt
R-06Fabricated Docker digest claim removed; honest TODO added pending network verificationHIGH✅ CLOSEDDockerfile
R-08secured_ci.yml SHA reference block stale for setup-pythonHIGH✅ CLOSED.github/workflows/secured_ci.yml
R-09secured_ci.yml SHA reference block stale for upload-artifactHIGH✅ CLOSED.github/workflows/secured_ci.yml
R-10Workflow permissions audit completed for secured_ci.yml, mermaid-render.yml, stale.yml, codeql.ymlMED✅ PARTIAL.github/workflows/*.yml
R-25mermaid-render.yml SHA reference block stale for setup-node; loop safety validatedLOW✅ CLOSED.github/workflows/mermaid-render.yml
CI-51stale.yml floating tag actions/stale@v9 SHA-pinned to verified commitMED✅ CLOSED.github/workflows/stale.yml
CI-52docs.yml SHA reference block stale for setup-python; synced to live pinLOW✅ CLOSED.github/workflows/docs.yml

Phase 3 — Architecture (Complete)

IDFindingSevStatusFiles Changed
ARCH-01HS256 symmetric JWT — upgrade to RS256 + JWKS rotationHIGH✅ CLOSEDsrc/auth/jwt_rs256.py; JWKS endpoint /.well-known/jwks.json
ARCH-02passlibargon2-cffi password hashing (OWASP 2024)HIGH✅ CLOSEDsrc/auth/password.py, requirements.txt
ARCH-03_USERS dict → PostgresUserRepository database-backedHIGH✅ CLOSEDsrc/users/repository.py, api/app.py, alembic/versions/0001_initial_schema.py
ARCH-04Secrets via Vault / AWS Secrets Manager (zero-secret images)HIGH✅ CLOSEDdocs/policies/secrets_management.md
ARCH-05GDPR /users/me/export and /users/me DELETE endpointsMED✅ CLOSEDapi/gdpr_routes.py
ARCH-06Argon2 rehash-on-login migration (zero-downtime)MED✅ CLOSEDsrc/auth/password.py, src/users/repository.py
ARCH-07Rate limiting: per-user sliding window via RedisMED✅ CLOSEDapi/rate_limit.py

Phase 2 — Infrastructure, Config, CI/CD (Complete)

IDFindingSevStatusFiles Changed
CI-06pip-audit had ‖ true bypass — CVE findings silently ignoredHIGH✅ CLOSEDci_cd/secure-ci.yml
CI-07No semgrep SAST — OWASP Top 10 pattern coverage gapMED✅ CLOSEDci_cd/secure-ci.yml
CI-08GitHub Actions not pinned to SHA digest — tag-mutation riskMED✅ CLOSEDci_cd/secure-ci.yml
CI-09Coverage threshold 70% — insufficient for security-critical auth pathsMED✅ CLOSEDci_cd/secure-ci.yml (raised to 75%)
CI-11No dependency-review action on PRsMED✅ CLOSEDci_cd/secure-ci.yml
CI-12Container scan built with ENVIRONMENT=test (misses production-mode issues)LOW✅ CLOSEDci_cd/secure-ci.yml
CI-13REDIS_PASSWORD not in CI test environmentMED✅ CLOSEDci_cd/secure-ci.yml
CI-14GitHub Actions workflow-level write-all permissionsMED✅ CLOSEDci_cd/secure-ci.yml (per-job least-privilege)
CFG-01No centralized Settings class — os.environ scattered across modulesMED✅ CLOSEDsrc/config.py (pydantic-settings + startup validation)
CFG-02.gitignore had unresolved merge conflict markersHIGH✅ CLOSED.gitignore
CFG-03.env.example missing REDIS_PASSWORD and DEV_*_PASSWORDMED✅ CLOSED.env.example

Phase 1 — Critical Security Hardening (Complete)

IDFindingSevStatusFiles Changed
P1-01noreply@ security contact replaced with GitHub Private Vulnerability ReportingHIGH✅ CLOSEDSECURITY.md
P1-02SECURITY.md controls table corrected to match actual repository stateMED✅ CLOSEDSECURITY.md
P1-03Airflow scope item removed — template copy-paste artifact, project has no AirflowLOW✅ CLOSEDSECURITY.md
P1-04All GitHub Actions workflows pinned to SHA digests (supply chain hardening)HIGH✅ CLOSED.github/workflows/ci.yml
P1-05Trivy gate restored to blocking exit-code: '1' (was bypassed as CI-26)HIGH✅ CLOSED.github/workflows/ci.yml
P1-06pip-audit JSON artifact generation decoupled from hard gate (removes silent failure)MED✅ CLOSED.github/workflows/ci.yml
P1-07CI secret availability guard added to integration-tests jobMED✅ CLOSED.github/workflows/ci.yml
P1-08CodeQL semantic SAST workflow added (security-and-quality query suite)HIGH✅ CLOSED.github/workflows/codeql.yml
CI-10Branch protection ruleset enforced on mainMED✅ CLOSEDGitHub repo Settings — Rulesets

Phase 12

ML-09 — Integration coverage gate raised 53% → 65% (CI-67 Complete)

Date: 2026-08-05
Files: tests/integration/test_inference_integration.py (new), .github/workflows/secured_ci.yml, Makefile, README.md

Problem: Integration --cov-fail-under gate was pinned at 53% (CI-66b deferral), while README claimed ≥65%. The inference router (api/routers/inference.py) had zero integration-level HTTP coverage. The mismatch was a direct factual contradiction visible in CI.

Fix: Added tests/integration/test_inference_integration.py with 10 ASGI-transport tests covering:

  • IT-INF-01..02: Happy path 200 (nominal + extreme vector)
  • IT-INF-03..04: 503 paths (missing artifact, predict() raises)
  • IT-INF-05: 401/403 unauthenticated gate
  • IT-INF-06: 422 Pydantic validation rejection
  • IT-INF-07..08: GET /anomaly/health (threshold exposed, artifact_exists reflected)
  • IT-INF-09: model_version constant round-trip
  • IT-INF-10: inference_latency_ms non-negative invariant

No Postgres or Redis required — ModelRegistry is patched per-test via unittest.mock.patch. Gate raised to 65% in both secured_ci.yml and Makefile. README Scope section updated to match.

Gate: --cov-fail-under=65


— ML Layer + Observability Hardening (2026-08-05)

IDFindingSevStatusFiles Changed
ML-01_ANOMALY_THRESHOLD was a hard-coded 0.0 constant; no env-var override; registry logs used stdlib logging not structlogHIGH✅ CLOSEDml_models/incident_anomaly/registry.py
ML-02scripts/train_model.py did not auto-generate SHA-256 artifact checksum; model_metadata.json missing artifact_sha256 and anomaly_threshold fieldsMED✅ CLOSEDscripts/train_model.py
ML-03api/routers/inference.py used stdlib logging.getLogger(__name__) — inference logs bypassed structlog PII-scrubbing and JSON pipelineHIGH✅ CLOSEDapi/routers/inference.py
ML-04MODEL_CARD.md Output Schema implied threshold was fixed; no threshold configuration guidance; confidence not annotated as uncalibratedMED✅ CLOSEDMODEL_CARD.md
ML-05No ADR covering anomaly model design decisions (algorithm selection, contamination rationale, threshold strategy, drift wiring)MED✅ CLOSEDdocs/adr/ADR-010-anomaly-model-design.md (new)
ML-06MODEL_CARD.md missing Productionization Gaps section; honest gap analysis not documentedLOW✅ CLOSEDMODEL_CARD.md
ML-07runbooks/model_degradation.md referenced drift detection generically without tying to implemented check_drift_suite(), ANOMALY_THRESHOLD env var, or API health endpointLOW✅ CLOSEDrunbooks/model_degradation.md
ML-08Dockerfile.dev base image python:3.11-slim not digest-pinned; no supply-chain rationale for intentional omissionLOW✅ CLOSEDDockerfile.dev

Phase 0 — Critical/High/Medium (Complete)

IDFindingSevStatusFiles Changed
CRIT-AHard-coded stub user passwords "admin-dev-only" silently used as fallbackCRIT✅ CLOSEDapi/app.py
CRIT-BAsync/sync denylist boundary: asyncio.get_event_loop().run_until_complete() inside async context raised RuntimeError on every revocationCRIT✅ CLOSEDsrc/redis_denylist.py, api/app.py ×3
CRIT-CAirflow + dbt dependencies in API image: +800 MB, arbitrary code exec surfaceCRIT✅ CLOSEDrequirements.txt, new requirements-airflow.txt
HIGH-ARedis: no AUTH, all-interface bind (0.0.0.0), no passwordHIGH✅ CLOSEDdocker-compose.yml, .env.example
HIGH-Bpython-jose CVE-2024-33663 (JWT algorithm confusion)HIGH✅ CLOSEDrequirements.txt (PyJWT), api/app.py
HIGH-CNo request body size limit — OOM DoS; no request timeout — slow-lorisHIGH✅ CLOSEDapi/middleware.py, api/app.py
HIGH-Dlru_cache on get_settings() causes env var bleed between testsHIGH✅ CLOSEDtests/conftest.py, src/config.py
MED-ADockerfile base image on floating tag (tag-mutation supply chain attack)MED✅ CLOSEDDockerfile (SHA-256 digest pinned on both stages)
MED-BFull repo .:/app bind mount; .dockerignore missingMED✅ CLOSEDdocker-compose.yml, .dockerignore
MED-CSlowAPI rate limiting missing on /auth/tokenMED✅ CLOSEDapi/app.py
MED-Dpasslib unmaintained; bcrypt unpinnedMED✅ CLOSEDrequirements.txt
MED-EMissing OWASP security headers (CSP, HSTS, X-Frame-Options, etc.)MED✅ CLOSEDapi/middleware.py
LOW-A.DS_Store tracked in git; merge conflicts in .gitignoreLOW✅ CLOSED.gitignore
LOW-BTelemetry ports bound to 0.0.0.0 (Jaeger UI, OTel receiver, Prometheus)LOW✅ CLOSEDdocker-compose.yml (loopback-only: 127.0.0.1:*)

Pre-Engagement Milestones

IDPhaseIssueResolutionDate
Phase 11Supply chain: lockfile-check CI job; pip-audit pre-commit; make deps-compile (CI-55)Complete2026-05-28
Phase 8Portfolio presentation: “Quick Proof of Quality” table in READMEPartial complete2026-05-27
Phase 7HF Spaces scaffolding: README YAML frontmatter; deploy-hf.yml workflowPartial complete2026-05-27
Phase 4Grafana + Prometheus infra: docker-compose.yml; dashboards/ml_operations_overview.jsonComplete2026-05-27
Phase 1Security hardening: Semgrep hard gate, CI_POSTGRES_PASSWORD secret ref, Bandit gate, per-job permissions:Complete (CI-52)2026-05-26
R-GODGod-file api/app.py (1005 lines) extracted into routers, middleware, auth, config modulesCompletePre-engagement

CI-10 Ruleset Detail — main branch (enforced 2026-05-24)

RuleSetting
Restrict deletions✅ Enabled
Require signed commits✅ Enabled
Require a PR before merging✅ Enabled
Required status checkssecrets-scan, dependency-audit, SAST - Bandit + mypy, test, 🧪 Tests (Python 3.11)
Require branches to be up to date✅ Enabled
Block force pushes✅ Enabled