Chapter 4: Persistence and Replication

Authored by syscook.dev

What are Persistence and Replication in Redis?

Persistence in Redis refers to the mechanisms that save data to disk to ensure durability, while replication involves creating copies of data across multiple Redis instances for high availability and load distribution. These features are crucial for production Redis deployments.

Key Concepts:

• RDB Persistence: Point-in-time snapshots of the dataset
• AOF Persistence: Append-only file logging of write operations
• Replication: Master-slave data synchronization
• High Availability: Automatic failover and recovery
• Data Durability: Ensuring data survives system failures
• Backup Strategies: Regular data backups and recovery

Why Use Persistence and Replication?

1. Data Durability

Persistence ensures your data survives system failures and restarts.

# RDB Configuration
# redis.conf
save 900 1    # Save if at least 1 key changed in 900 seconds
save 300 10   # Save if at least 10 keys changed in 300 seconds
save 60 10000 # Save if at least 10000 keys changed in 60 seconds

# AOF Configuration
appendonly yes
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

Persistence Benefits:

• Data survives system crashes
• Point-in-time recovery
• Efficient backup and restore
• Minimal performance impact

2. High Availability

Replication provides redundancy and automatic failover capabilities.

# Master configuration
# redis.conf
bind 127.0.0.1
port 6379
requirepass master_password

# Slave configuration
# redis-slave.conf
bind 127.0.0.1
port 6380
replicaof 127.0.0.1 6379
masterauth master_password

Replication Benefits:

• Automatic failover
• Load distribution
• Data redundancy
• Geographic distribution

3. Scalability

Replication enables read scaling and load distribution.

# Multiple slaves for read scaling
replicaof 127.0.0.1 6379
replicaof 127.0.0.1 6379
replicaof 127.0.0.1 6379

# Load balancing across slaves
# Application can read from multiple slaves

How to Configure Persistence and Replication?

1. RDB Persistence Configuration

Basic RDB Setup

# redis.conf
# Snapshotting configuration
save 900 1
save 300 10
save 60 10000

# RDB file configuration
dbfilename dump.rdb
dir /var/lib/redis

# RDB compression and checksums
rdbcompression yes
rdbchecksum yes

# Background save behavior
stop-writes-on-bgsave-error yes

RDB Operations

# Manual RDB save
redis-cli BGSAVE

# Check last save time
redis-cli LASTSAVE

# Get RDB file information
redis-cli CONFIG GET dbfilename
redis-cli CONFIG GET dir

# Disable automatic saves
redis-cli CONFIG SET save ""

2. AOF Persistence Configuration

Basic AOF Setup

# redis.conf
# AOF configuration
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec

# AOF rewrite configuration
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

# AOF load configuration
aof-load-truncated yes
aof-use-rdb-preamble yes

AOF Operations

# Manual AOF rewrite
redis-cli BGREWRITEAOF

# Check AOF file size
redis-cli CONFIG GET appendfilename
ls -la /var/lib/redis/appendonly.aof

# AOF file repair
redis-check-aof --fix appendonly.aof

# Disable AOF
redis-cli CONFIG SET appendonly no

3. Replication Configuration

Master Configuration

# redis-master.conf
bind 127.0.0.1
port 6379
requirepass master_password

# Replication settings
repl-diskless-sync no
repl-diskless-sync-delay 5
repl-ping-replica-period 10
repl-timeout 60
repl-disable-tcp-nodelay no
repl-backlog-size 1mb
repl-backlog-ttl 3600

# Logging
logfile /var/log/redis/redis-master.log

Slave Configuration

# redis-slave.conf
bind 127.0.0.1
port 6380
replicaof 127.0.0.1 6379
masterauth master_password

# Slave settings
replica-serve-stale-data yes
replica-read-only yes
repl-diskless-load disabled

# Logging
logfile /var/log/redis/redis-slave.log

Replication Commands

# Check replication status
redis-cli INFO replication

# Manual replication setup
redis-cli REPLICAOF 127.0.0.1 6379

# Stop replication
redis-cli REPLICAOF NO ONE

# Check master-slave connection
redis-cli ROLE

Practical Examples

1. Complete Replication Setup

Master-Slave Configuration

#!/bin/bash
# setup-replication.sh

# Start master
redis-server --port 6379 --requirepass master_password --daemonize yes

# Start slave
redis-server --port 6380 --replicaof 127.0.0.1 6379 --masterauth master_password --daemonize yes

# Test replication
redis-cli -p 6379 -a master_password SET test_key "test_value"
redis-cli -p 6380 -a master_password GET test_key

# Check replication status
redis-cli -p 6379 -a master_password INFO replication
redis-cli -p 6380 -a master_password INFO replication

echo "Replication setup completed!"

Multiple Slaves Setup

#!/bin/bash
# setup-multiple-slaves.sh

# Start master
redis-server --port 6379 --requirepass master_password --daemonize yes

# Start multiple slaves
redis-server --port 6380 --replicaof 127.0.0.1 6379 --masterauth master_password --daemonize yes
redis-server --port 6381 --replicaof 127.0.0.1 6379 --masterauth master_password --daemonize yes
redis-server --port 6382 --replicaof 127.0.0.1 6379 --masterauth master_password --daemonize yes

# Test replication to all slaves
redis-cli -p 6379 -a master_password SET test_key "test_value"

# Check all slaves
redis-cli -p 6380 -a master_password GET test_key
redis-cli -p 6381 -a master_password GET test_key
redis-cli -p 6382 -a master_password GET test_key

echo "Multiple slaves setup completed!"

2. Persistence Configuration

RDB and AOF Setup

#!/bin/bash
# setup-persistence.sh

# Create Redis configuration with both RDB and AOF
cat > redis-persistence.conf <<EOF
# RDB Configuration
save 900 1
save 300 10
save 60 10000
dbfilename dump.rdb
dir /var/lib/redis
rdbcompression yes
rdbchecksum yes
stop-writes-on-bgsave-error yes

# AOF Configuration
appendonly yes
appendfilename "appendonly.aof"
appendfsync everysec
no-appendfsync-on-rewrite no
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb
aof-load-truncated yes
aof-use-rdb-preamble yes

# General Configuration
bind 127.0.0.1
port 6379
daemonize yes
logfile /var/log/redis/redis-persistence.log
EOF

# Start Redis with persistence
redis-server redis-persistence.conf

# Test persistence
redis-cli SET test_key "test_value"
redis-cli BGSAVE
redis-cli BGREWRITEAOF

echo "Persistence setup completed!"

3. Backup and Recovery

Backup Script

#!/bin/bash
# backup-redis.sh

BACKUP_DIR="/backup/redis"
DATE=$(date +%Y%m%d_%H%M%S)
REDIS_DIR="/var/lib/redis"

# Create backup directory
mkdir -p $BACKUP_DIR

# Create RDB backup
redis-cli BGSAVE
sleep 5
cp $REDIS_DIR/dump.rdb $BACKUP_DIR/dump_$DATE.rdb

# Create AOF backup
cp $REDIS_DIR/appendonly.aof $BACKUP_DIR/appendonly_$DATE.aof

# Compress backups
gzip $BACKUP_DIR/dump_$DATE.rdb
gzip $BACKUP_DIR/appendonly_$DATE.aof

# Clean old backups (keep last 7 days)
find $BACKUP_DIR -name "*.gz" -mtime +7 -delete

echo "Backup completed: $BACKUP_DIR"

Recovery Script

#!/bin/bash
# restore-redis.sh

BACKUP_FILE=$1
REDIS_DIR="/var/lib/redis"

if [ -z "$BACKUP_FILE" ]; then
    echo "Usage: $0 <backup_file>"
    exit 1
fi

# Stop Redis
redis-cli SHUTDOWN

# Restore RDB file
cp $BACKUP_FILE $REDIS_DIR/dump.rdb

# Start Redis
redis-server --daemonize yes

echo "Recovery completed from: $BACKUP_FILE"

Best Practices

1. Persistence Configuration

# Use both RDB and AOF for maximum durability
save 900 1
save 300 10
save 60 10000
appendonly yes
appendfsync everysec

# Optimize AOF rewrite
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

2. Replication Setup

# Use strong passwords
requirepass strong_password_here
masterauth strong_password_here

# Configure appropriate timeouts
repl-timeout 60
repl-ping-replica-period 10

# Set appropriate backlog size
repl-backlog-size 1mb

3. Monitoring and Maintenance

# Monitor replication lag
redis-cli INFO replication

# Check persistence status
redis-cli INFO persistence

# Monitor memory usage
redis-cli INFO memory

Common Pitfalls and Solutions

1. Replication Lag Issues

# ❌ Problem: High replication lag
# Solution: Check network and configuration

# Check replication status
redis-cli INFO replication

# Check network connectivity
ping master_host

# Adjust timeout settings
repl-timeout 60
repl-ping-replica-period 10

2. Persistence Performance Issues

# ❌ Problem: Slow persistence operations
# Solution: Optimize configuration

# Use appropriate sync frequency
appendfsync everysec

# Optimize AOF rewrite
auto-aof-rewrite-percentage 100
auto-aof-rewrite-min-size 64mb

3. Data Loss Issues

# ❌ Problem: Data loss after restart
# Solution: Ensure proper persistence

# Enable both RDB and AOF
save 900 1
appendonly yes
appendfsync everysec

# Test persistence
redis-cli SET test_key "test_value"
redis-cli BGSAVE
redis-cli SHUTDOWN
redis-server
redis-cli GET test_key

Conclusion

Persistence and replication are essential for production Redis deployments. By understanding:

• What persistence and replication mechanisms are available
• Why they're important for data durability and high availability
• How to configure and manage them effectively

You can build robust Redis deployments that ensure data safety and provide high availability. Proper configuration of these features is crucial for production environments where data loss is not acceptable.

Next Steps

• Practice with different persistence and replication scenarios
• Learn about Redis clustering and high availability
• Move on to Chapter 5: Clustering and High Availability

This tutorial is part of the Redis Mastery series by syscook.dev