How do I roll back a bad deployment?
Manual rollback steps
If you deploy with a versioned artifact (Docker image tag, JAR, ZIP), redeploy the previous version. For example, with Docker Swarm: `docker service update --rollback <service>`. With Kubernetes: `kubectl rollout undo deployment/<name>`. With systemd: copy the old binary back and restart the service.
If you use a blue-green setup, switch traffic back to the blue environment. If you use feature flags, turn off the bad flag. Always verify the rollback with a health check or smoke test.
- Identify the last good version (tag, commit SHA, build number).
- Run the platform rollback command or redeploy that version.
- Check logs and metrics to confirm the service is healthy.
- Notify your team and document the incident.
Automating rollbacks
In CI/CD, add a rollback step that runs automatically if health checks fail after deployment. For example, in a Jenkins pipeline, after deploying, run a script that curls a health endpoint; if it returns non-200, trigger `kubectl rollout undo`.
Keep the last few artifacts available (e.g., in a registry) so rollback is fast. Some platforms like Kubernetes keep a revision history (default 10) that you can roll back to with `kubectl rollout undo --to-revision=2`.
Common mistakes
- Assuming rollback is instant; database migrations or state changes may not be reversible without extra work.
- Not testing the rollback procedure before you need it.
- Rolling back the code but forgetting to revert configuration or environment variables.
