Reliability, resilience, and disaster recovery
Availability keeps a service working. Disaster recovery restores it after a serious event. Backup is only one recovery ingredient.
Start with business numbers
- RTO: maximum acceptable time to restore service.
- RPO: maximum acceptable amount of lost data measured in time.
- SLO: measurable reliability target, such as successful requests.
Do not buy a multi-Region design before the business agrees it needs the complexity and cost.
Reliability patterns
- Distribute across AZs and remove single points of failure.
- Scale horizontally and keep compute replaceable.
- Buffer bursty work with queues.
- Use timeouts, bounded retries, exponential backoff, and jitter.
- Make operations idempotent.
- Use health checks, graceful degradation, and load shedding.
- Test backups and recovery, not only backup creation.
- Practice failure with game days.
DR strategies
| Strategy | Relative cost | Recovery |
|---|---|---|
| Backup and restore | low | slowest |
| Pilot light | low-medium | core services always ready |
| Warm standby | medium-high | reduced-size environment running |
| Multi-site active/active | highest | fastest, most complex |
Multi-AZ versus Multi-Region
Multi-AZ handles data-center/AZ failure and should be the normal availability baseline for important regional workloads. Multi-Region addresses larger regional or sovereignty needs but adds data replication, consistency, routing, deployment, testing, and operational complexity.
Common trap
Retries can cause an outage if every caller retries together. Bound retries, back off with jitter, make requests safe to repeat, and stop when the remaining time cannot support another attempt.