🔍

When performing backups and data transfers with Rsync, data corruption and inconsistency can occur due to network issues, disk failures, or interrupted transfers.

Rsync offers powerful checksum verification features to ensure that files are accurately transferred and not corrupted. This guide will teach you how to verify data integrity before and after Rsync transfers.

📌 In this guide, you will learn:
✅ How Rsync uses checksums to detect file changes
✅ How to verify data integrity after transfers
✅ How to force Rsync to re-check and sync only corrupted files
✅ How to compare source and destination files for consistency

🛑 1. Why Data Integrity Matters in Rsync?

🔹 Silent Data Corruption – Bit errors can occur without obvious signs.
🔹 Interrupted Transfers – A network drop can cause partial file copies.
🔹 Hardware Failures – Disk errors can lead to data corruption.
🔹 Unreliable Storage Devices – Some USB drives and cloud storage may cause data inconsistencies.

✅ Solution: Use Rsync checksums and verification tools to detect and fix inconsistencies.

⚡ 2. Understanding Rsync Checksum Verification

🔹 2.1 How Rsync Uses Checksums

Rsync uses checksum verification to detect file changes and integrity issues.

✅ Key options for checksum verification:

• --checksum (-c) → Forces Rsync to compare checksums instead of timestamps.
• --ignore-existing → Ensures Rsync does not overwrite already existing files.
• --dry-run → Tests Rsync without making changes.

✅ Rsync checksum verification process:
1️⃣ Rsync reads the source and destination files.
2️⃣ Rsync calculates a checksum (MD4/MD5) for each file.
3️⃣ If the checksum differs, Rsync re-transfers the file.

📌 This ensures that corrupted or modified files are detected and fixed.

🔎 3. Verifying Rsync File Integrity

🔹 3.1 Running Rsync with Checksum Verification

✅ Use --checksum (-c) to verify file integrity:

rsync -avc /source/ user@remote:/backup/

📌 This forces Rsync to re-check each file using checksums instead of timestamps.
📌 Slower but ensures all files are correctly transferred.

✅ If a file is corrupted or changed, Rsync will re-transfer it.

🔹 3.2 Comparing Source and Destination Files

✅ Use rsync --dry-run to check for differences:

rsync -av --dry-run /source/ user@remote:/backup/

📌 Simulates the transfer without modifying files.

✅ Use diff to compare two directories:

diff -qr /source/ /backup/

📌 -q shows only the names of differing files.

✅ Use find to check for corrupted files:

find /backup/ -type f -exec md5sum {} + > backup_checksums.txt

📌 Compares MD5 checksums of backup files.

🔹 3.3 Re-Syncing Corrupted Files

If some files are incomplete or corrupted, force Rsync to fix them.

✅ Use --ignore-existing to sync only missing files:

rsync -av --ignore-existing /source/ user@remote:/backup/

📌 Ensures that Rsync does not overwrite existing valid files.

✅ Use --checksum and --delete-missing to remove bad files and re-sync them:

rsync -avc --delete-missing-args /source/ user@remote:/backup/

📌 Deletes files that failed checksum verification and re-syncs them.

🛠️ 4. Automating Rsync Integrity Checks

For regular backups, you should automate integrity verification.

🔹 4.1 Create a Verification Script

✅ Save the following script as /usr/local/bin/rsync-verify.sh:

#!/bin/bash

# Directories
SOURCE_DIR="/source/"
BACKUP_DIR="/backup/"

# Log file
LOG_FILE="/var/log/rsync_verify.log"

# Perform checksum-based Rsync sync
rsync -avc --log-file=$LOG_FILE "$SOURCE_DIR" "$BACKUP_DIR"

# Compare source and backup for inconsistencies
diff -qr "$SOURCE_DIR" "$BACKUP_DIR" >> $LOG_FILE

echo "Rsync integrity verification completed!"

✅ Make the script executable:

chmod +x /usr/local/bin/rsync-verify.sh

✅ Schedule it to run daily:

crontab -e

✅ Add the following line:

0 3 * * * /usr/local/bin/rsync-verify.sh

📌 Runs Rsync verification every night at 3 AM.

⚠️ 5. Troubleshooting Rsync Integrity Issues

✅ Debug Rsync checksum failures:

rsync -avc --progress /source/ user@remote:/backup/

📌 --progress shows real-time file sync details.

✅ Check Rsync logs for integrity issues:

tail -f /var/log/rsync_verify.log

📌 Logs reveal which files failed verification.

📊 6. Summary

✅ Using Rsync checksums and verification ensures your backups remain accurate and corruption-free.

💬 Join the Discussion!

How do you verify data integrity in Rsync?
Do you use checksums or other methods for ensuring file accuracy?

💬 Share your experience in the comments below! 🚀

👉 Next Up: Using Rsync with ZFS & Btrfs for Snapshot Backups