Canary/Blue-Green Deployment
What This Requires
Deploy model updates using canary or blue-green strategy: route small percentage of traffic to new version, monitor error rates and latency, auto-rollback if metrics degrade, gradually increase traffic if stable.
Why It Matters
Big-bang deployments amplify the impact of bugs. Canary deployments limit blast radius and enable quick detection of issues before full rollout.
How To Implement
Choose Strategy
Canary: route 5% traffic to new version, monitor for 1 hour, increase to 50%, then 100%. Blue-Green: deploy new version alongside old, switch 100% traffic atomically after validation.
Define Success Metrics
Monitor: error rate (<1%), latency p99 (<500ms), user-reported issues. Auto-rollback if any metric exceeds threshold.
Traffic Routing
Use load balancer (ALB, NGINX) or service mesh (Istio, Linkerd) to split traffic. Configure gradual rollout (5% → 25% → 50% → 100% over 4 hours).
Automated Rollback
Integrate monitoring with deployment tool (Spinnaker, Argo Rollouts). Trigger rollback if error rate spikes or latency degrades.
Evidence & Audit
- Canary deployment policy with traffic ramp schedule
- Success metrics and rollback thresholds defined
- Traffic routing configuration (load balancer, service mesh)
- Deployment logs showing gradual rollout and monitoring
- Auto-rollback test results