Demo 4: ODF Regional-DR

Version 0.6.2 · draft · 2026-10-16 · class/demos/04-odf-regional-dr.md

What you will learn

You will protect running virtual machines with OpenShift Data Foundation (ODF) Regional-DR, which replicates their disks asynchronously from site A to site B every 15 minutes. You will prove that a replication cycle really completed, by injecting rows into PostgreSQL and watching them reach the other site. You will move the VMs to site B and home again without losing a single row, and you will see that only the changes travel, not whole disks. In the bonus round you break the replication link, force a failover and measure exactly what an unplanned disaster costs (the RPO).

How the lab is set up for this demo

How the two clusters are joined (ACM hub, Submariner, then ODF's MirrorPeer/DRPolicy), what this lab has today and what CI automates: Lab S §S.14. Read it first: this demo depends on all three layers.

The booking's CI customization (the dr-fabric and dr-policy jobs, which run when DR is on) built all of this before class. See the lab diagram, and Lab S — how this lab was built for the CLI and console steps behind the shared pieces.

Pre-built	Where	How to check	How it was built · Red Hat docs
ODF on BOTH sites (storage: odf on site A and site B, an operator toggle on the booking)	site A, site B	oc --context site-X -n openshift-storage get storagecluster → Ready on both	S.4 · ODF 4.19
ACM and cert-manager on both clusters (demo-openshift's base build, the same operators on both). DR runs from site B's hub, with site A imported as a managed cluster	site B	oc --context site-b get managedclusters	S.10 · ACM 2.16
ODF Multicluster Orchestrator and the DR hub operator (Ramen) on the hub, and the DR cluster operator on both sites	site B / both	§4.2	ODF DR guide
Submariner (the network tunnel Ceph mirrors over; Globalnet and the provider-server export only if the sites' IPv4 ranges overlap), the MirrorPeer, both DRClusters	both	§4.2	ODF DR guide: Regional-DR
The same VLAN bridges on both sites: every vmnet<VLAN> your VMs use exists on site A and site B, same name, same VLAN. A Relocate or Failover brings back the VM's own definition, which names its bridge (default/vmnet23): without that bridge on the other site the VM doesn't start, and with it the VM keeps its address	both	oc --context site-a -n default get net-attach-def and the same on site-b: the same vmnet* names	demo-openshift's VLAN build on each site (BL-B29)
The 15-minute DRPolicy (sync_interval=15m)	site B	§4.2	§4.2
Your namespace with your browser terminal sql-client and your VM's Service (<VM>-sql); ovedr-app with the instructor's, on both sites	both	oc --context site-b -n "$VMNS" get deploy,svc	S.9
Your own VM ($VM, e.g. win-pg-s03) running on site A, on ODF storage (ocs-storagecluster-ceph-rbd-virtualization), at its own address $VM_IP on the VLAN bridge (Demo 1's staged plan)	site A	where → site-a Running	S.11 · S.12
Your Windows login (.\<WINUSER>) and the PostgreSQL toolkit (marks, ovedr_*) inside your VM	your VM	verify	S.11

Two identical sites, and the one step CI runs across both

CI's customization builds site A and site B the same, each site on its own: migration tools, the vSphere provider and maps, the staged plans d1-<vm> / d2-<vm>, your namespace and sql-client, the VLAN bridges. That's why Demo 1 can fall back to site B when site A has a problem (Demo 1 §1.3).

The ODF DR clustering is the one workflow that can't run on each site in parallel, because it joins the two: demo-prep imports site A into site B's ACM hub, then dr-fabric turns on the object gateway (MCG) on both sites, links their networks with Submariner, and pairs the two Ceph clusters (MirrorPeer → both DRClusters), and dr-policy adds the 15-minute DRPolicy. It runs once, after both sites are built, from the hub. Lab S §S.14 has each step as code.

This demo starts from site A, the first deploy. Your VM is protected where Demo 1 put it, on site A, and replicates to B. If you fell back to site B in Demo 1, tell your instructor before §4.0: they halt your site B copy (both sites share your VLAN and address) and re-run your site A plan, so where shows site-a Running.

Who is in charge: the hub supervises, the sites are not peers

Two clusters, but they don't have equal roles. One ACM hub supervises both. Only the hub can protect a VM, run a Relocate or a Failover, or show the whole picture. In this lab the hub is on site B (ocp-lz2 in the examples).

Cluster	Its role	What its console's All Clusters → Clusters shows
Site B (ocp-lz2)	the hub (ACM, ODF Multicluster Orchestrator, Ramen's DR hub operator), and a managed cluster (local-cluster), and the DR target	both clusters, Ready
Site A (ocp-lz1)	a managed cluster only: runs the VMs, the DR cluster operator, and mirrors its disks to B	nothing, by design: ACM lets a cluster belong to only one hub, and site A's belongs to site B's

What the hub knows while a site is down. A hub can't see a cluster that is down. It shows that cluster as Unknown and keeps what it last heard: each protected namespace's DR state and its last completed sync. What failover needs is already on the surviving site: the disks' snapshots, mirrored there every interval, and the VM and volume definitions, which the DR operators keep in an object store on each site [verify: Ramen's S3 metadata store on ODF 4.19].

What fails	Can you fail over?	What to do
Site A (the VMs' home)	✅ yes: the hub on B is up	Failover to B from the hub (§4.8). When A is back, the hub sees it again, reverses replication, and you relocate home (§4.6). This is why the hub sits on B.
Site B (the hub's site)	❌ no: nothing can drive a failover	The VMs on A keep running, but replication to B stops and no Relocate or Failover can run. Bring the hub back online first: once it's up, it reconnects to A, replication resumes, and you can manage the other cluster again.
The hub, for good	❌ not until a hub exists again	Restore the hub's backup onto another cluster (ACM's hub backup and restore, with a passive hub kept ready), then carry on from it [verify].

⚠ A lab compromise. Red Hat's supported Regional-DR layout has three clusters: the hub on its own, away from both sites, usually with a passive standby hub. Then losing either site never costs you the hub. This lab has two clusters, so the hub shares site B. That's fine for the disaster this demo shows (site A lost), but the first step of any recovery is getting the hub back: it's the cluster you manage all the others from. ACM: hub backup and restore [verify link].

What you do yourself: inspect every DR layer, inject and verify data at every stage, read replication status at three levels, and run the moves when you are the driver.

⚠ Capacity, stated plainly. Each ODF cluster takes all 4 scratch SSDs, one on each demosvr host. Today the lab has 4, so it can run one ODF site. Every other demo runs with site A on ODF and site B on NFS. Demo 4 needs a booking with ODF on both sites, which becomes possible when the additional drives are installed. Until then, run this demo from the 📸 screenshots only.

One protection unit for the whole class. Each student's VM lives in that student's own namespace (their class ID), the instructor's in ovedr-app. The default is one DRPC protecting all of them: ovedr-app plus every student namespace [verify: a discovered-app DRPC with several protectedNamespaces]. Every Relocate or Failover moves every student's VM together, so 🧑‍🏫 the driver runs the DR actions, and each of you verifies your own VM. Per-VM protection (one DRPC per student VM, selecting that VM's disks by label) is possible but not built here [verify].

Before you start: do Lab 0, then sign your class terminal in on both sites and to your database (Toolkit T.1-T.2: oclogin for site A and site B, then dblogin). It already holds the variables ($ME, $N, $A, $B, $VM, $VM_IP, $WINUSER) and the commands (sql, mark, load, verify, alive, where).

🖥️ Where you type the commands: your browser terminal. Every bash block in this guide runs in your own terminal pod, in the browser: in the site A console, project <your class ID> → Workloads → Pods → sql-client → Terminal (Toolkit T.1). Nothing is installed on your computer. It already knows your VM and its address (type ovedr); dblogin and oclogin (Toolkit T.1-T.2) stay signed in while the pod runs. Variables you export last until you close that Terminal tab: if you reopen it, re-run the guide's export lines. Paste with right-click → Paste (or Ctrl+Shift+V / ⌘V). powershell blocks run inside your Windows VM, from its console (vCenter Launch Web Console, or the VM's Console tab in OpenShift).

_{🎬 Demo 4: ODF Regional-DR: a walk-through of the whole demo in the console. Watch it, then do the same with the steps below.}

---

4.0 Point the helpers at your VM

Use your VM's own address, $VM_IP, for every connection test. Your VM sits on the VLAN bridge (vmnet<VLAN>), which exists with the same name on both sites, so after a Relocate or Failover it comes up with the same address on the other site. (Only a VM on the pod network gets a new pod IP per site: then refresh it with Toolkit T.4 after every move.) Your VM's Service ($VM-sql) also follows the VM between sites.

export PGHOST=$VM_IP                                  # or: export PGHOST=$VM-sql
export DRPC=ovedr-app                                 # the class's one DRPC (ovedr-app + every student namespace)
# the DR state of the class's protected namespace, read from the hub (site B)
drpc() { oc --context site-b -n openshift-dr-ops get drpc "$DRPC" \
           -o jsonpath='{"phase="}{.status.phase}{"  progression="}{.status.progression}{"  lastSync="}{.status.lastGroupSyncTime}{"  dur="}{.status.lastGroupSyncDuration}{"  bytes="}{.status.lastGroupSyncBytes}{"\n"}'; }
# Ceph's own view of mirroring, on one site (needs rsh into the tools pod: instructor screen if your role cannot)
rbdmirror() { oc --context "$1" -n openshift-storage rsh deploy/rook-ceph-tools \
                rbd mirror pool status ocs-storagecluster-cephblockpool --verbose; }

where; alive
mark d4-start
verify

✍️ Row 4.0: your d4-start mark id and the fingerprint.

---

4.1 Inside the VM: your file

In your VM's console on site A (Virtualization → VirtualMachines → <your VM> → Console), log in as .\<WINUSER>, then run:

Add-Content "C:\Users\Public\ovedr\$env:USERNAME.txt" "$(Get-Date -Format o)  d4-start on site A"
Get-Content "C:\Users\Public\ovedr\$env:USERNAME.txt" -Tail 3

📸 04-touch-start: the last three lines of your file, on site A.

---

4.2 Set up asynchronous replication between the clusters

Regional-DR is built in layers. The lab stages the lower ones (dr-fabric); you inspect each one and see what it is for.

Layer	Object (on the hub unless noted)	What it does
Cluster set	ManagedClusterSet, both ManagedClusters	ACM knows both sites
Network	Submariner Broker + add-on on each cluster	a tunnel between the clusters' networks, which carries Ceph's mirror traffic
Storage peering	MirrorPeer (ODF Multicluster Orchestrator)	turns on RBD mirroring between the two Ceph clusters and exchanges their keys and S3 stores
DR clusters	DRCluster ×2 (created by the MirrorPeer)	Ramen's view of each site and its S3 store
Policy	DRPolicy	which two clusters pair up, and how often to checkpoint

oc --context site-b get managedclusters                                   # ocp-lzX, ocp-lzY (+ local-cluster): JOINED, AVAILABLE
oc --context site-b get managedclusteraddons -A | grep -E 'submariner|NAME'
oc --context site-b get mirrorpeer -o custom-columns=NAME:.metadata.name,TYPE:.spec.type,PHASE:.status.phase
oc --context site-b get drcluster \
  -o custom-columns=NAME:.metadata.name,S3:.spec.s3ProfileName,VALIDATED:'.status.conditions[?(@.type=="Validated")].status'

Submariner links the two sites' pod and service networks. If the sites have separate ranges (the plan), it runs without Globalnet. If only their IPv4 ranges overlap, it runs with Globalnet, and each site exports ODF's provider server so the two Ceph clusters can find each other (Lab S §S.14 layer 2):

oc --context site-b -n ovedr-broker get broker submariner-broker -o jsonpath='{.spec.globalnetEnabled}{"\n"}'   # false (true only for an IPv4 overlap)
for s in site-a site-b; do
  echo "== $s"
  oc --context $s -n openshift-storage get storageclusterpeer -o custom-columns=NAME:.metadata.name,STATE:.status.state   # Peered
  oc --context $s -n openshift-storage get cephblockpoolradosnamespaces ocs-storagecluster-cephblockpool-builtin-implicit \
    -o jsonpath='{.status.mirroringStatus.summary}{"\n"}'                   # "daemon_health":"OK", "health":"OK"
done

Example code: how the lab built these layers (instructor)

The full commands, layer by layer, with the console path for each: Lab S §S.14. CI runs them as dr_fabric.sh (Submariner, MCG, the provider-server export when Globalnet is on, the Orchestrator, the MirrorPeer) and dr_policy.sh (the 15-minute DRPolicy) on every booking with ODF on both sites.

Debug: a layer is missing or unhealthy

What you see	Look at	Fix
no mirrorpeer or drcluster at all	the booking's dr-fabric job	the booking isn't ODF on both sites (storage: odf), or dr-fabric failed: its log names the step
Submariner add-on not AVAILABLE	oc --context site-b -n <cluster> get managedclusteraddon submariner -o yaml	Lab S §S.14 layer 2 debug: gateway node label, UDP 4500 between the gateway nodes
storageclusterpeer missing or not Peered	oc --context site-X -n openshift-storage get serviceexport ocs-provider-server -o yaml	the provider-server export and StorageCluster annotation (Lab S §S.14 layer 3, step 2)
mirroring health WARNING	wait: a new link takes up to 10 minutes	still WARNING/ERROR after that: Lab S §S.14 layer 3 debug

🧑‍🏫 Driver: the 15-minute DRPolicy

The user-facing promise of this demo is "a checkpoint every 15 minutes". That is the schedulingInterval.

export POLICY=$A-$B-15m
cat <<YAML | oc --context site-b apply -f -
apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPolicy
metadata: { name: $POLICY }
spec:
  drClusters: [ $A, $B ]
  schedulingInterval: 15m
  replicationClassSelector: {}
  volumeSnapshotClassSelector: {}
YAML

The interval must be one the MirrorPeer offers (spec.schedulingIntervals). The lab's MirrorPeer offers 2m, 5m and 15m, and CI has usually created this policy already (dr_policy.sh), so the apply just confirms it.

Debug: the DRPolicy stays validated=False

oc --context site-b get drpolicy "$POLICY" -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}: {.message}{"\n"}{end}'
oc --context site-b get mirrorpeer -o jsonpath='{.items[0].spec.schedulingIntervals}{"\n"}'      # must include 15m
oc --context site-b get drcluster -o custom-columns=NAME:.metadata.name,VALIDATED:'.status.conditions[?(@.type=="Validated")].status'

An interval the MirrorPeer doesn't offer, or a drClusters name that isn't a DRCluster (ocp-<A>, local-cluster), are the usual causes. One DRPolicy per interval; you can't edit an existing policy's interval (Red Hat ODF 4.19).

Check

oc --context site-b get drpolicy "$POLICY" \
  -o jsonpath='{.spec.schedulingInterval}{"  validated="}{.status.conditions[?(@.type=="Validated")].status}{"\n"}'
# the per-interval replication classes the MirrorPeer created on each site
for s in site-a site-b; do echo "== $s"; oc --context $s get volumereplicationclass; done

✅ 15m validated=True. ✅ Each site has a VolumeReplicationClass whose schedulingInterval is 15m.

📸 04-drpolicy: the DRPolicy validated at 15m.

---

4.3 Protect the VM

mark d4-before-protect

🧑‍🏫 Driver: enrol the VM

The VMs were created by MTV, not by a GitOps application, so they are a discovered application. Protect ovedr-app (the instructor's VM and sql-client) and every student namespace (each student's VM, its disks and its Service), so a move carries everything the class uses.

Console, site B (hub): All Clusters → Data Services → Disaster recovery → Protected applications → Enroll discovered application [verify: BL-T07]: - cluster $A - namespaces ovedr-app and every student-NN-<booking> namespace - policy $POLICY - resources: all in namespace - kube-object capture interval 15m

That creates the Placement and the DRPlacementControl (DRPC) on the hub. As YAML, roughly [verify]:

apiVersion: ramendr.openshift.io/v1alpha1
kind: DRPlacementControl
metadata: { name: ovedr-app, namespace: openshift-dr-ops }
spec:
  preferredCluster: <A>
  drPolicyRef: { name: <A>-<B>-15m }
  placementRef: { kind: Placement, name: ovedr-app-placement, namespace: openshift-dr-ops }
  protectedNamespaces: [ ovedr-app, student-01-<booking>, student-02-<booking> ]   # one entry per student namespace [verify]
  pvcSelector: {}
  kubeObjectProtection: { captureInterval: 15m, kubeObjectSelector: {} }

Check: the first sync is a FULL copy

oc --context site-b -n openshift-dr-ops get drpc "$DRPC" -o wide -w          # Ctrl-C at Deployed / Protected
drpc
oc --context site-a -n "$VMNS" get volumereplicationgroup                 # one VRG, state Primary
oc --context site-a -n "$VMNS" get volumereplicationgroup "$DRPC" \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

✅ The VRG reports DataReady=True, DataProtected=True and ClusterDataProtected=True [verify: condition names on ODF 4.19]. ✍️ Row 4.3: the first lastGroupSyncBytes and lastGroupSyncDuration. This is the only full copy in the demo. Compare it with the VM's disk usage (oc --context site-a -n "$VMNS" get pvc).

📸 04-protected: the Protected applications page showing ovedr-app Healthy.

---

4.4 Prove replication is running, and that a cycle completed

What "running" looks like, at three levels

drpc                                                           # 1 · Ramen: the last group sync
oc --context site-a -n "$VMNS" get volumereplication          # 2 · CSI: one VolumeReplication per disk
oc --context site-a -n "$VMNS" get volumereplication \
  -o custom-columns=PVC:.spec.dataSource.name,STATE:.status.state,LASTSYNC:.status.lastSyncTime
rbdmirror site-a                                               # 3 · Ceph: images up+stopped (primary), health OK
rbdmirror site-b                                               #     images up+replaying (secondary)

To see one image's mirror snapshots, which are the checkpoints themselves:

oc --context site-a -n "$VMNS" get pvc | grep "$VM"              # your VM's three disks: pick one
PVC='<one of your VM pvcs>'
IMG=$(oc --context site-a get pv "$(oc --context site-a -n "$VMNS" get pvc "$PVC" -o jsonpath='{.spec.volumeName}')" \
        -o jsonpath='{.spec.csi.volumeAttributes.imageName}')
oc --context site-a -n openshift-storage rsh deploy/rook-ceph-tools \
  rbd mirror image status "ocs-storagecluster-cephblockpool/$IMG"
oc --context site-a -n openshift-storage rsh deploy/rook-ceph-tools \
  rbd snap ls --all "ocs-storagecluster-cephblockpool/$IMG" | grep -i mirror

✅ Primary: state: up+stopped, with description showing the last mirror snapshot. Secondary: up+replaying.

Inject, then wait for ONE complete sync that contains your row

load d4-busy 200                                 # a burst: the database is busy
mark d4-wait-for-sync                            # the row that must reach site B
T=$(sql -Atc "SELECT max(ts) FROM marks WHERE student = '$ME'")
echo "my row's time: $T"
# poll the hub until a group sync STARTED after your row's time (each sync is a crash-consistent snapshot)
until [ "$(date -d "$(oc --context site-b -n openshift-dr-ops get drpc "$DRPC" -o jsonpath='{.status.lastGroupSyncTime}')" +%s 2>/dev/null || echo 0)" \
        -gt "$(date -d "$T" +%s)" ]; do
  printf '%s  waiting… ' "$(date +%T)"; drpc; sleep 60
done
echo "✅ a sync completed after your row"; drpc

The loop uses GNU date -d, which your browser terminal (Linux) has.

💡 With a 15-minute interval this can take up to 15 minutes. That wait is the RPO you are buying. The instructor may use a 5-minute policy in a short session.

✅ lastGroupSyncTime is later than your row, so your d4-wait-for-sync row is now on site B's disks. ✍️ Row 4.4: a typical checkpoint's bytes and dur. Compare them with row 4.3: it should be a small fraction.

📸 04-sync-complete: the drpc line after the loop ends.

---

4.5 Snapshot, then a planned failover to site B

A snapshot as a local safety net

cat <<YAML | oc --context site-a apply -f -
apiVersion: snapshot.kubevirt.io/v1beta1
kind: VirtualMachineSnapshot
metadata: { name: $VM-d4-pre-move, namespace: $VMNS }
spec:
  source: { apiGroup: kubevirt.io, kind: VirtualMachine, name: $VM }
YAML
oc --context site-a -n "$VMNS" get vmsnapshot "$VM-d4-pre-move" -w        # Ctrl-C at readyToUse=true

[verify] snapshot.kubevirt.io/v1beta1 is the served version on OpenShift Virtualization 4.19 (v1alpha1 on older releases).

💡 A snapshot is not DR. It lives on the same Ceph cluster as the VM. It protects against a bad change (Demo 1's bonus round), not against losing the site. Replication is what protects the site.

Relocate vs Failover

	Relocate (planned)	Failover (unplanned)
When	both sites healthy: maintenance, evacuation, a test	the primary is gone or unreachable
Order	stop the VM on A → final sync → demote A → promote B → start on B	promote B from the last completed sync → start on B
Data loss	none	up to one interval (the RPO)

A "clean migration" between sites is a Relocate. The bonus round does a real Failover.

🧑‍🏫 Driver: Relocate to site B

mark d4-before-relocate
oc --context site-b -n openshift-dr-ops patch drpc "$DRPC" --type merge \
  -p "{\"spec\":{\"action\":\"Relocate\",\"preferredCluster\":\"$B\"}}"

(Console: Protected applications → ovedr-app → ⋮ → Relocate.)

Check: on site B, nothing lost

Your VM came up on site B with the same address ($PGHOST), so the helpers work unchanged:

oc --context site-b -n openshift-dr-ops get drpc "$DRPC" -o wide -w           # Relocating → Relocated
where                                                                        # site-b Running
oc --context site-b -n "$VMNS" get vm,vmi,pvc,svc,deploy                   # everything came with it
alive
verify
sql -c "SELECT id, ts, stage FROM marks WHERE student = '$ME' ORDER BY id DESC LIMIT 5;"

✅ d4-before-relocate is present on site B, so a planned move loses nothing. ✅ chain_ok = t. The fingerprint equals the one you would get on A at the moment it stopped. ✍️ Row 4.5: Relocate duration (DRPC lastGroupSyncDuration, plus the heartbeat gap from alive) and rows lost (0).

Your file, in the console on site B:

Get-Content "C:\Users\Public\ovedr\$env:USERNAME.txt" -Tail 3        # d4-start is there
Add-Content "C:\Users\Public\ovedr\$env:USERNAME.txt" "$(Get-Date -Format o)  d4 on site B"

📸 04-on-site-b: where showing site-b, and the file's tail.

---

4.6 Reverse replication B → A, then come home cleanly

The VM now runs on B, and B is the primary. Before you move it again, check that protection has reversed: B replicates back to A, and A's images are the secondaries.

mark d4-on-b
drpc
oc --context site-b -n openshift-dr-ops get drpc "$DRPC" \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'
rbdmirror site-b            # B: up+stopped (primary)
rbdmirror site-a            # A: up+replaying (secondary): the reverse direction is live

✅ PeerReady=True and Protected=True. ✅ A's images are up+replaying. Wait for one sync after d4-on-b, using the same loop as §4.4, with site-b in place of site-a in the sql call.

🧑‍🏫 Driver: Relocate home to site A

mark d4-before-home
oc --context site-b -n openshift-dr-ops patch drpc "$DRPC" --type merge \
  -p "{\"spec\":{\"action\":\"Relocate\",\"preferredCluster\":\"$A\"}}"
oc --context site-b -n openshift-dr-ops get drpc "$DRPC" -o wide -w

Check: the VM is home, and the data made the full round trip

Same address again, now on site A:

where                                            # site-a Running
alive
verify
sql -c "SELECT stage, count(*), min(id), max(id) FROM marks WHERE student = '$ME' GROUP BY stage ORDER BY min(id);"
drpc                                             # Relocated; protection A → B again

✅ Every stage is present: d4-start, d4-busy (200 rows), d4-wait-for-sync, d4-before-protect, d4-before-relocate, d4-on-b and d4-before-home. ✅ chain_ok = t. ✍️ Row 4.6: the bytes of the return sync. It is a delta (what changed on B), not the whole disk.

Console on site A:

Get-Content "C:\Users\Public\ovedr\$env:USERNAME.txt" -Tail 4        # both "site A" and "site B" lines

📸 04-home: verify with chain_ok = t, and the file's tail.

---

4.7 End-of-demo validation

alive
verify
sql -c "SELECT * FROM ovedr_verify();"                        # the whole class: every student's chain_ok
sql -c "SELECT id, ts, date_trunc('second', gap) AS outage FROM (
                 SELECT id, ts, ts - lag(ts) OVER (ORDER BY id) AS gap FROM heartbeat) g
               WHERE gap > interval '10 seconds' ORDER BY id DESC LIMIT 5;"   # each move's outage (RTO)

Row	Stage	Bytes	Duration	Rows lost	Your last mark id	chain_ok	Fingerprint
4.0	start	—	—	—
4.3	first sync (full)			—
4.4	typical 15-min checkpoint			—
4.5	Relocate A → B			0
4.6	Relocate B → A (home)			0

The claim holds if rows 4.4, 4.5 and 4.6 are each a small fraction of row 4.3, and every planned move lost zero rows.

🧑‍🏫 Optional cleanup: oc --context site-a -n "$VMNS" delete vmsnapshot "$VM-d4-pre-move".

---

Thank you

Thank you for working through Demo 4. You protected live VMs across two clusters, proved that each checkpoint reached the DR site, and moved the whole application there and back with every row intact. To go further:

• Lab S — how this lab was built: the shared pieces, from the CLI and in the console
• ODF disaster recovery for OpenShift workloads (4.19): https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.19/html/configuring_openshift_data_foundation_disaster_recovery_for_openshift_workloads/index
• Red Hat OpenShift Data Foundation 4.19 documentation: https://docs.redhat.com/en/documentation/red_hat_openshift_data_foundation/4.19
• Red Hat Advanced Cluster Management 2.13: https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.13
• OpenShift Virtualization (OpenShift 4.19): https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html/virtualization/index
• Training: DO316, Managing Virtual Machines with Red Hat OpenShift Virtualization: https://www.redhat.com/en/services/training/do316-managing-virtual-machines-red-hat-openshift-virtualization
• Red Hat interactive labs: https://www.redhat.com/en/interactive-labs
• OpenShift Data Foundation: https://www.redhat.com/en/technologies/cloud-computing/openshift-data-foundation

Stay for the bonus round if time allows. It shows what the 15-minute interval costs when the link fails without warning.

---

4.8 BONUS: a hostile failover, with a broken replication link

Now the link fails, the business can't wait, and the class forces the VM onto site B from whatever the last completed sync holds. Then it repairs the link and forces the resync back.

How to break the link: options

Option	What happens	Realism	Risk	Recovery
A. Cut the Submariner tunnel (recommended) 🧑‍🏫	Ceph mirror traffic stops. Both clusters stay up, and ACM still talks to both	"the WAN link died"	low: one label on one node	put the label back
B. Power off all site-A nodes 🧑‍🏫 (virsh destroy, instructor guide §4.1)	site A vanishes entirely	"the site burned down"	medium: Ceph on A restarts from a crash	virsh start; wait for Ceph HEALTH_OK
C. Scale down rbd-mirror on site B	the secondary stops pulling	narrow: a storage fault	rook/ocs operators scale it back up unless they are paused too	scale the operators back up
D. Fence site A (DRCluster spec.clusterFence: Fenced)	Metro-DR only. Regional-DR does not fence [verify]	—	—	not used

Recommended: option A. It matches "break the ODF sync link" exactly: replication fails while both sites still run, so the class sees a link failure rather than a site loss. It is reversed with one command. Option B is the scripted disaster in the classic M6 module.

Step 1: inject, and note the last sync that got through

mark d4h-before-break
until [ "$(date -d "$(oc --context site-b -n openshift-dr-ops get drpc "$DRPC" -o jsonpath='{.status.lastGroupSyncTime}')" +%s)" \
        -gt "$(date -d "$(sql -Atc "SELECT max(ts) FROM marks WHERE student='$ME'")" +%s)" ]; do drpc; sleep 60; done
drpc                       # ✍️ lastSync = the last checkpoint that will reach B
sql -Atc "SELECT max(id) FROM marks WHERE student = '$ME'"          # ✍️ your last SAFE id

Step 2: 🧑‍🏫 break the link (option A) [verify]

# the gateway node(s) on site B carry the label submariner.io/gateway=true; removing it stops the tunnel
GW=$(oc --context site-b get nodes -l submariner.io/gateway=true -o name)
echo "$GW" > /tmp/ovedr-gw-site-b.txt
oc --context site-b label $GW submariner.io/gateway-
oc --context site-b -n submariner-operator get gateways -o wide      # no active gateway / connection error

Step 3: keep writing on A. These rows will NOT reach B.

for n in 1 2 3; do mark "d4h-after-break-$n"; sleep 20; done
sql -c "SELECT id, ts, stage FROM marks WHERE student = '$ME' ORDER BY id DESC LIMIT 4;"

✍️ Write down these ids and the first 12 characters of each sha. They are the rows you are about to lose.

# in your VM's console on site A
Add-Content "C:\Users\Public\ovedr\$env:USERNAME.txt" "$(Get-Date -Format o)  d4h written AFTER the link broke"

After one interval, the hub notices that syncs have stopped:

drpc                                                    # lastSync stops advancing
oc --context site-b -n openshift-dr-ops get drpc "$DRPC" \
  -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'

Step 4: 🧑‍🏫 force Failover to site B

oc --context site-b -n openshift-dr-ops patch drpc "$DRPC" --type merge \
  -p "{\"spec\":{\"action\":\"Failover\",\"failoverCluster\":\"$B\"}}"
oc --context site-b -n openshift-dr-ops get drpc "$DRPC" -o wide -w        # FailingOver → FailedOver

ACM still reaches site A, so Ramen stops the VM there (it becomes the secondary) and starts it on B from the last completed sync.

Step 5: measure what was lost (the RPO)

Same address, on site B:

where                                                   # site-b Running
verify
sql -c "SELECT id, ts, stage, left(sha,12) AS sha FROM marks WHERE student = '$ME' ORDER BY id DESC LIMIT 4;"

What you should see, and why: - The last row present is your last SAFE id from step 1. Every d4h-after-break-* row is missing. - chain_ok is still t. A chain detects rows lost, changed or reordered in the middle. Losing the tail leaves a shorter chain that is still valid. That is why you wrote down your ids: your notes are the evidence of what the failover cost. - Write a new mark on B. Its id may reuse one of the lost ids, because B's sequence continues from the replicated state. Its sha differs from your note, so it is a different row with the same number.

mark d4h-on-b-after-failover
sql -c "SELECT id, stage, left(sha,12) FROM marks WHERE student = '$ME' ORDER BY id DESC LIMIT 2;"

# console on site B: the "AFTER the link broke" line is gone, because the disk is from the last sync
Get-Content "C:\Users\Public\ovedr\$env:USERNAME.txt" -Tail 3

✍️ Row 4.8a: rows lost (count of d4h-after-break-*), RPO (failover time minus the last lastSync), and the outage (heartbeat gap on B).

📸 04-hostile-lost-rows: your noted ids beside B's marks tail.

Step 6: 🧑‍🏫 restore the link and force the resync

oc --context site-b label $(cat /tmp/ovedr-gw-site-b.txt) submariner.io/gateway=true
oc --context site-b -n submariner-operator get gateways -o wide       # connected again
drpc                                                                  # progression moves on

If progression stops at WaitOnUserToCleanUp, the stale VM on site A must go before A can become the secondary. That is the instructor's cleanup, in instructor guide §4.3 [verify: BL-T09].

Then watch A's images become secondaries and catch up from the last common snapshot:

rbdmirror site-a          # A: syncing → up+replaying
rbdmirror site-b          # B: up+stopped (primary)

[verify: BL-T09] If an image stays in split-brain or error, Ceph needs an explicit resync of that image on the old primary (A). This copies the whole image, so use it only when Ramen does not recover by itself:

oc --context site-a -n openshift-storage rsh deploy/rook-ceph-tools \
  rbd mirror image resync "ocs-storagecluster-cephblockpool/<image>"

✍️ Row 4.8b: resync bytes and duration. A delta (from the last common checkpoint) proves the DESIGN's claim. A full copy means the hostile path costs one full copy, and the guide says so honestly (DESIGN §6.4).

Step 7: 🧑‍🏫 Relocate home, and verify the history

mark d4h-before-home
# wait for one sync after that mark (§4.4 loop, site-b), then:
oc --context site-b -n openshift-dr-ops patch drpc "$DRPC" --type merge \
  -p "{\"spec\":{\"action\":\"Relocate\",\"preferredCluster\":\"$A\"}}"
oc --context site-b -n openshift-dr-ops get drpc "$DRPC" -o wide -w
where
verify
sql -c "SELECT id, stage, left(sha,12) AS sha FROM marks WHERE student = '$ME' AND stage LIKE 'd4h%' ORDER BY id;"

✅ Present: d4h-before-break, d4h-on-b-after-failover and d4h-before-home. ✅ Missing: every d4h-after-break-*. These were written only on A after the link broke, then discarded when A was resynced from B. That is exactly the data the hostile failover cost. ✅ chain_ok = t on A. The chain continues from the last safe row, through B, and home.

Row	Stage	Bytes	Duration	Rows lost	Notes
4.8a	forced Failover A ✗ → B	—			RPO =
4.8b	resync after the link returns			—	delta or full?
4.8c	Relocate home			0

📸 04-hostile-home: the final d4h% query on site A.

---

Glossary

Term	Meaning
Regional-DR	asynchronous replication between two clusters. RPO is greater than 0, at most one schedulingInterval
MirrorPeer	ODF Multicluster Orchestrator's pairing of two Ceph clusters, which turns on RBD mirroring
DRPolicy	the two clusters, and the checkpoint interval (here 15m)
DRPC	DRPlacementControl: the DR state of one protected workload, and where you ask for Relocate or Failover
VRG	VolumeReplicationGroup: the per-cluster agent that makes the workload's disks primary or secondary
Relocate	planned move: final sync, zero loss
Failover	forced move from the last completed sync: loss up to one interval
RPO / RTO	how much data you can lose / how long you are down