---
title: "Reliability, resilience, and disaster recovery"
chapter: "09"
---

# Reliability, resilience, and disaster recovery

Availability keeps a service working. Disaster recovery restores it
after a serious event. Backup is only one recovery ingredient.

## Start with business numbers

- **RTO:** maximum acceptable time to restore service.
- **RPO:** maximum acceptable amount of lost data measured in time.
- **SLO:** measurable reliability target, such as successful requests.

Do not buy a multi-Region design before the business agrees it needs the
complexity and cost.

## Reliability patterns

- Distribute across AZs and remove single points of failure.
- Scale horizontally and keep compute replaceable.
- Buffer bursty work with queues.
- Use timeouts, bounded retries, exponential backoff, and jitter.
- Make operations idempotent.
- Use health checks, graceful degradation, and load shedding.
- Test backups and recovery, not only backup creation.
- Practice failure with game days.

## DR strategies

| Strategy | Relative cost | Recovery |
|---|---:|---|
| Backup and restore | low | slowest |
| Pilot light | low-medium | core services always ready |
| Warm standby | medium-high | reduced-size environment running |
| Multi-site active/active | highest | fastest, most complex |

## Multi-AZ versus Multi-Region

Multi-AZ handles data-center/AZ failure and should be the normal availability
baseline for important regional workloads. Multi-Region addresses larger
regional or sovereignty needs but adds data replication, consistency, routing,
deployment, testing, and operational complexity.

## Common trap

Retries can cause an outage if every caller retries together. Bound retries,
back off with jitter, make requests safe to repeat, and stop when the remaining
time cannot support another attempt.