🛡 Resilience: Backup, DR & Chaos
Overview
The Resilience area brings four related practices together in one place: automated backup policy generation, disaster recovery (DR) drills, chaos engineering experiments, and environment teardown verification. Each capability produces a timestamped, persisted report so you always have evidence of your last test. The goal is to make recovery rehearsals a routine part of development rather than a scramble after an incident.
How it works
Each capability is backed by a dedicated .NET service on the backend. The BackupPolicyGenerator derives a policy (cron schedule, retention window, encryption and off-region settings) from your chosen cloud and data-store type, then records a BackupRun with its status in the database. The DrDrillOrchestrator shifts traffic from a primary region to a standby via ITrafficShifter, probes replication lag to measure RPO, times the cutover to measure RTO, then fails traffic back — the drill status advances through TrafficShifting → StandbyActive → Failback → Completed. The ChaosExperimentRunner generates a ready-to-apply template (Chaos Mesh YAML, AWS FIS JSON, or Azure Chaos Studio JSON) for the chosen fault type and target, then polls a steady-state probe to record whether the system stayed healthy and how many incidents were triggered. The DestroyProofGenerator calls a Terraform destroy adapter and then individually checks each expected cloud resource to confirm it is absent, producing a per-resource verification table and screenshots path. Restore drills run a backup's data into an isolated environment and validate row count and checksum before marking the proof Validated or Failed.
What you can do
How to use it
- Open the DR panel for your project
In the Oryn desktop, navigate to your project and open the Resilience / DR section. You will see tabs for Backup, Restore Drill, DR Drill, Chaos Experiment, and Destroy Proof. - Generate and run a backup policy
In the Backup tab, choose your cloud (Azure, AWS, or GCP) and data-store type (SQL, Blob, or Queue), then click Run backup. Oryn generates a policy with the appropriate cron schedule and retention window (for example, hourly with 35-day retention for SQL), records a BackupRun, and displays it in the history table below. - Verify the backup with a restore drill
In the Restore Drill tab, paste the backup run ID from the history table, give the isolated environment a name (e.g. restore-iso-1), enter the expected row count and checksum, then click Run restore drill. Oryn restores the data into that isolated environment and compares actual rows and checksum against your expectations, reporting Validated or Failed. - Run a DR failover drill
In the DR Drill tab, enter your primary and standby region identifiers (e.g. us-east-1 and us-west-2), set your RPO and RTO targets in seconds, then click Run DR drill. Oryn shifts traffic to the standby, measures replication lag for RPO and cutover time for RTO, fails traffic back, and shows colour-coded gauges indicating whether each target was met. - Inject a chaos fault
In the Chaos Experiment tab, select a tool (Chaos Mesh, AWS FIS, or Azure Chaos Studio), choose a fault type (Pod Kill, Network Latency, Disk Fill, or CPU Stress), enter a target selector (e.g. app=web), set duration in seconds, and click Run experiment. Oryn generates the deployment template and probes whether steady state was preserved after the fault, then shows the result alongside the exportable template. - Produce a destroy proof for an ephemeral environment
In the Destroy Proof tab, enter the environment tag (e.g. ephemeral-pr-42) and paste a JSON list of expected resources with their type, ID, and region. Click Run destroy proof. Oryn destroys the environment via Terraform and then checks each resource individually, reporting which are confirmed absent and which are still present, plus a screenshots path for audit evidence.