Cross-Region Replication¶

Replicate validation results across multiple storage backends for high availability and disaster recovery.

Overview¶

The replication module provides:

Multi-region replication - Sync data across geographic regions
Multiple modes - Sync, async, and semi-sync replication
Read preferences - Route reads to optimal replicas
Conflict resolution - Handle concurrent writes
Health monitoring - Track replica health and lag
Automatic failover - Switch to healthy replicas on failure

Replication Modes¶

from truthound.stores.replication.base import ReplicationMode

ReplicationMode.SYNC       # Wait for all replicas
ReplicationMode.ASYNC      # Fire and forget
ReplicationMode.SEMI_SYNC  # Wait for at least N replicas

Mode	Consistency	Latency	Use Case
`SYNC`	Strong	High	Critical data
`ASYNC`	Eventual	Low	High throughput
`SEMI_SYNC`	Bounded	Medium	Balanced

Quick Start¶

from truthound.stores import get_store
from truthound.stores.replication.base import (
    ReplicationConfig,
    ReplicationMode,
    ReplicaTarget,
)

# Create primary and replica stores
primary = get_store("s3", bucket="primary-bucket", region="us-east-1")
replica1 = get_store("s3", bucket="replica-bucket", region="us-west-2")
replica2 = get_store("s3", bucket="replica-bucket", region="eu-west-1")

# Configure replication
config = ReplicationConfig(
    mode=ReplicationMode.ASYNC,
    targets=[
        ReplicaTarget(
            name="us-west-2",
            store=replica1,
            region="us-west-2",
            priority=1,
        ),
        ReplicaTarget(
            name="eu-west-1",
            store=replica2,
            region="eu-west-1",
            priority=2,
        ),
    ],
)

ReplicaTarget¶

Define each replication target:

from truthound.stores.replication.base import ReplicaTarget, ReplicaHealth

target = ReplicaTarget(
    name="us-west-2",                  # Unique name
    store=replica_store,               # Store backend
    region="us-west-2",                # Geographic region
    priority=1,                        # Failover priority (lower = higher)
    is_read_replica=False,             # Read-only replica
    sync_timeout_seconds=30.0,         # Sync operation timeout
    max_retry_attempts=3,              # Max retries on failure
    retry_delay_seconds=1.0,           # Base delay between retries
    health=ReplicaHealth.UNKNOWN,      # Initial health status
)

ReplicaTarget Fields¶

Field	Type	Default	Description
`name`	`str`	required	Unique replica name
`store`	`ValidationStore`	required	Store backend
`region`	`str`	required	Geographic region
`priority`	`int`	`1`	Failover priority
`is_read_replica`	`bool`	`False`	Read-only replica
`sync_timeout_seconds`	`float`	`30.0`	Sync timeout
`max_retry_attempts`	`int`	`3`	Max retry attempts
`retry_delay_seconds`	`float`	`1.0`	Retry delay

Replica Health Management¶

# Mark health status
target.mark_healthy()
target.mark_degraded(lag_ms=5000)
target.mark_unhealthy()
target.mark_syncing()

# Pause/resume replication
target.pause()
target.resume()

Configuration¶

ReplicationConfig¶

from truthound.stores.replication.base import (
    ReplicationConfig,
    ReplicationMode,
    ReadPreference,
    ConflictResolution,
)

config = ReplicationConfig(
    mode=ReplicationMode.SEMI_SYNC,
    targets=[...],
    read_preference=ReadPreference.NEAREST,
    conflict_resolution=ConflictResolution.LAST_WRITE_WINS,
    min_sync_replicas=1,               # For semi-sync mode
    enable_health_checks=True,
    health_check_interval_seconds=30.0,
    max_replication_lag_ms=5000.0,
    enable_failover=True,
    enable_metrics=True,
)

# Validate configuration
config.validate()

Configuration Options¶

Option	Type	Default	Description
`mode`	`ReplicationMode`	`ASYNC`	Replication mode
`targets`	`list[ReplicaTarget]`	`[]`	Replica targets
`read_preference`	`ReadPreference`	`PRIMARY`	Read routing
`conflict_resolution`	`ConflictResolution`	`LAST_WRITE_WINS`	Conflict strategy
`min_sync_replicas`	`int`	`1`	Min replicas for semi-sync
`enable_health_checks`	`bool`	`True`	Enable health monitoring
`health_check_interval_seconds`	`float`	`30.0`	Health check interval
`max_replication_lag_ms`	`float`	`5000.0`	Max acceptable lag
`enable_failover`	`bool`	`True`	Enable auto failover
`enable_metrics`	`bool`	`True`	Collect metrics

Read Preferences¶

Control where reads are routed:

from truthound.stores.replication.base import ReadPreference

ReadPreference.PRIMARY     # Always read from primary
ReadPreference.SECONDARY   # Prefer secondary replicas
ReadPreference.NEAREST     # Read from nearest replica
ReadPreference.ANY         # Read from any available

Preference	Consistency	Latency	Use Case
`PRIMARY`	Strong	Higher	Critical reads
`SECONDARY`	Eventual	Lower	Read scaling
`NEAREST`	Eventual	Lowest	Geo-distributed
`ANY`	Eventual	Variable	High availability

Conflict Resolution¶

Handle concurrent writes to multiple regions:

from truthound.stores.replication.base import ConflictResolution

ConflictResolution.LAST_WRITE_WINS   # Most recent write wins
ConflictResolution.FIRST_WRITE_WINS  # First write wins
ConflictResolution.PRIMARY_WINS      # Primary always wins
ConflictResolution.MERGE             # Custom merge logic
ConflictResolution.MANUAL            # Require manual resolution

Health Status¶

from truthound.stores.replication.base import ReplicaHealth, ReplicaState

# Health status
ReplicaHealth.HEALTHY     # Up and synchronized
ReplicaHealth.DEGRADED    # Up but lagging
ReplicaHealth.UNHEALTHY   # Down or unreachable
ReplicaHealth.UNKNOWN     # Status unknown

# Replica state
ReplicaState.ACTIVE       # Actively receiving writes
ReplicaState.PAUSED       # Temporarily paused
ReplicaState.SYNCING      # Catching up after failure
ReplicaState.FAILED       # Replication failed
ReplicaState.REMOVED      # Removed from replication

Replication Metrics¶

Track replication performance:

from truthound.stores.replication.base import ReplicationMetrics

metrics = ReplicationMetrics()

# Record operations
metrics.record_primary_write()
metrics.record_replication_success("us-west-2", time_ms=50.0)
metrics.record_replication_failure("eu-west-1")
metrics.record_conflict(resolved=True)
metrics.record_failover()
metrics.update_lag("us-west-2", lag_ms=100.0)

# Get statistics
print(f"Primary writes: {metrics.writes_to_primary}")
print(f"Replicated: {metrics.writes_replicated}")
print(f"Failed: {metrics.writes_failed}")
print(f"Conflicts: {metrics.conflicts_detected}")
print(f"Failovers: {metrics.failovers}")

# Success rate per replica
rate = metrics.get_replica_success_rate("us-west-2")
print(f"US-West-2 success rate: {rate:.1f}%")

# Average replication time
avg_time = metrics.get_average_replication_time()
print(f"Avg replication time: {avg_time:.2f}ms")

# Full metrics dict
metrics_dict = metrics.to_dict()

ReplicationMetrics Fields¶

Field	Description
`writes_to_primary`	Total writes to primary
`writes_replicated`	Successful replications
`writes_failed`	Failed replications
`conflicts_detected`	Conflict count
`conflicts_resolved`	Resolved conflicts
`failovers`	Failover count
`current_lag_ms`	Current lag per replica

Real-World Examples¶

Multi-Region HA Setup¶

config = ReplicationConfig(
    mode=ReplicationMode.SEMI_SYNC,
    min_sync_replicas=1,  # At least 1 replica must ack
    targets=[
        ReplicaTarget(
            name="us-west-2",
            store=get_store("s3", bucket="bucket-west", region="us-west-2"),
            region="us-west-2",
            priority=1,
        ),
        ReplicaTarget(
            name="eu-west-1",
            store=get_store("s3", bucket="bucket-eu", region="eu-west-1"),
            region="eu-west-1",
            priority=2,
        ),
    ],
    read_preference=ReadPreference.NEAREST,
    enable_failover=True,
    max_replication_lag_ms=5000.0,
)

Read Scaling¶

config = ReplicationConfig(
    mode=ReplicationMode.ASYNC,
    targets=[
        ReplicaTarget(
            name="read-replica-1",
            store=read_store_1,
            region="us-east-1",
            is_read_replica=True,
        ),
        ReplicaTarget(
            name="read-replica-2",
            store=read_store_2,
            region="us-east-1",
            is_read_replica=True,
        ),
    ],
    read_preference=ReadPreference.SECONDARY,
)

Disaster Recovery¶

config = ReplicationConfig(
    mode=ReplicationMode.SYNC,  # Strong consistency
    targets=[
        ReplicaTarget(
            name="dr-site",
            store=dr_store,
            region="us-west-2",
            priority=1,
            sync_timeout_seconds=60.0,
            max_retry_attempts=5,
        ),
    ],
    conflict_resolution=ConflictResolution.PRIMARY_WINS,
    enable_health_checks=True,
    health_check_interval_seconds=10.0,
)

Best Practices¶

Sync Mode¶

Use for critical data requiring strong consistency
Accept higher latency for guaranteed durability
Monitor replica health closely

Async Mode¶

Use for high-throughput scenarios
Accept eventual consistency
Monitor replication lag

Semi-Sync Mode¶

Balance between consistency and latency
Set min_sync_replicas based on durability requirements
Use with at least 2 replicas

Health Monitoring¶

# Check health before operations
for target in config.targets:
    if target.health == ReplicaHealth.UNHEALTHY:
        # Alert or take action
        pass

# Monitor lag
for target in config.targets:
    if target.replication_lag_ms > config.max_replication_lag_ms:
        target.mark_degraded(target.replication_lag_ms)

Next Steps¶

Observability - Audit, metrics, tracing
Caching - In-memory caching layer
Tiering - Hot/Warm/Cold/Archive storage

Failure and shutdown contract¶

Synchronous and semi-synchronous save failure raises StoreWriteError after a primary write may already exist. Do not blindly retry it. Call save_async inside an event loop; synchronous save rejects that usage before a primary write. flush and stop drain instance-owned async tasks and queued writes; counters report replica failures, and the queue is not durable. Failed batch/full synchronization preserves success=False; failed full sync marks the target unhealthy. An optional trusted set_io_guard(callback) checks immediately before primary/replica I/O, including background reads and writes. No callback is serialized. MCP store contracts expose these native operations through fixed aliases and exact approval.

The native replication monitor awaits sync or async event handlers in registration order and attempts remaining handlers after one fails. Handler failures raise an ExceptionGroup from direct emission and are recorded by background sweeps; callbacks are not automatically retried for the same health transition. Failed health checks mark that replica unhealthy, preserve the configured interval and allow other replicas to be checked. get_health_summary() exposes running, cumulative monitor_failures/handler_failures and the last 100 phase/replica/handler-index/timestamp records without exception messages. stop() cancels and awaits its owned task. MCP replication_stats includes bounded monitor errors with public replica aliases, without private names. Intervals and lag limits must be finite.