Optimizing Rsync Performance for Large Files & Datasets

CloudNetOps

14 Feb 2025 — 3 min read

🚀

When dealing with large files, huge datasets, or millions of small files, a standard Rsync setup can be slow and inefficient. Transfers may take hours or even days, causing network congestion and high CPU usage.

This guide provides step-by-step instructions to optimize Rsync performance for high-speed, large-scale data synchronization.

📌 In this guide, you will learn:
✅ How to speed up Rsync for large files & millions of small files
✅ How to use parallelization & multi-threading for faster transfers
✅ How to limit Rsync bandwidth to prevent network overload
✅ How to efficiently resume interrupted large file transfers

🛑 1. Challenges of Rsync with Large Files & Datasets

Rsync is great for incremental synchronization, but it struggles with:

🔹 Large Single Files – Gigabyte or terabyte-sized files slow down transfers.
🔹 Millions of Small Files – File indexing and metadata updates can be extremely slow.
🔹 Inefficient Bandwidth Use – Rsync can consume all available bandwidth, affecting other applications.
🔹 Interrupted Transfers – If a transfer fails mid-way, Rsync starts over by default, wasting time.

✅ Solution: Apply Rsync optimizations to accelerate large-scale file transfers.

⚡ 2. Enabling Rsync Compression for Faster Transfers

By default, Rsync copies files without compression, consuming more bandwidth.

✅ Use the -z flag to enable compression:

rsync -avz /data/ user@backup-server:/backups/

📌 Breakdown of options:

-a → Preserve timestamps, permissions, symbolic links.
-v → Enable verbose output.
-z → Enable gzip compression for faster transfers.

✅ Check transfer speed with and without compression:

time rsync -av /data/ user@backup-server:/backups/
time rsync -avz /data/ user@backup-server:/backups/

📌 Expected Improvement:

Without compression: Transfer time = 30 min
With compression: Transfer time = 18 min

📌 Best Use Case: Compression is most effective for text-based files but has minimal impact on already compressed files like .zip, .mp4, .tar.gz.

📂 3. Handling Millions of Small Files More Efficiently

When transferring millions of tiny files, Rsync spends more time scanning directories and reading file metadata than transferring actual data.

✅ Optimize Rsync to improve small file transfers:

rsync -av --info=progress2 --delete /data/ user@backup-server:/backups/

📌 Why?

--info=progress2 → Shows per-file transfer statistics.
--delete → Removes files that no longer exist at the source (avoids unnecessary scans).

✅ For extreme cases, use find to generate a file list first:

find /data/ -type f -print0 | rsync -av --files-from=- --from0 / user@backup-server:/backups/

📌 Why?

find efficiently lists files before Rsync starts, reducing scanning overhead.
-print0 and --from0 handle filenames with spaces properly.

✅ Measure Performance Gains:

time rsync -av /data/ user@backup-server:/backups/
time find /data/ -type f -print0 | rsync -av --files-from=- --from0 / user@backup-server:/backups/

📌 Expected Speed Improvement:

Standard Rsync: 4 hours
Optimized Rsync: 1.5 hours

📶 4. Using Parallel Transfers for Faster Rsync

By default, Rsync processes files sequentially, making it slow for large datasets.

✅ Use GNU Parallel to enable multi-threaded Rsync:

find /data/ -type f | parallel -j4 rsync -avz {} user@backup-server:/backups/

📌 Why?

parallel -j4 runs 4 Rsync processes in parallel, dramatically speeding up transfers.

✅ Monitor Rsync CPU usage:

htop

📌 Check if multiple Rsync processes are running.

✅ Performance Boost for Large Files:

time rsync -avz /data/ user@backup-server:/backups/
time find /data/ -type f | parallel -j4 rsync -avz {} user@backup-server:/backups/

📌 Expected Improvement:

Single-threaded Rsync: 2 hours
Parallel Rsync: 45 minutes

⏳ 5. Resuming Large File Transfers Efficiently

If Rsync fails mid-transfer, it restarts from the beginning by default.

✅ Enable resume functionality using --partial:

rsync -avz --partial /data/ user@backup-server:/backups/

📌 Allows Rsync to pick up from where it left off instead of re-transferring files.

✅ Use --append for even faster resumption:

rsync -avz --append /data/ user@backup-server:/backups/

📌 --append ensures Rsync continues where the last transfer ended.

🌐 6. Limiting Bandwidth to Avoid Network Overload

By default, Rsync consumes all available bandwidth, potentially slowing down other applications.

✅ Use --bwlimit to cap Rsync’s bandwidth usage:

rsync -avz --bwlimit=5000 /data/ user@backup-server:/backups/

📌 Limits Rsync to 5 MB/s, preventing network congestion.

✅ Test different bandwidth limits:

rsync -avz --bwlimit=1000 /data/ user@backup-server:/backups/  # 1MB/s
rsync -avz --bwlimit=5000 /data/ user@backup-server:/backups/  # 5MB/s

📌 Measure how each setting affects other network activity.

📊 7. Monitoring and Debugging Rsync Performance

🔹 7.1 Check Rsync Transfer Statistics

✅ Use --progress to track Rsync progress:

rsync -avz --progress /data/ user@backup-server:/backups/

📌 Expected Output:

data/file1.txt  100%  45MB/s  00:05
data/file2.txt  100%  42MB/s  00:04

🔹 7.2 Log Rsync Transfer Time

✅ Measure Rsync execution time:

time rsync -avz /data/ user@backup-server:/backups/

📌 Expected Output:

real 10m45s
user 5m30s
sys 1m15s

📌 Shows total time taken for Rsync to complete.

🔹 7.3 Analyze Rsync Network Usage

✅ Monitor Rsync bandwidth consumption:

sudo iftop -i eth0

📌 Shows real-time network usage of Rsync transfers.

📊 8. Summary

Optimization	Command	Use Case
Compression (`-z`)	`rsync -avz`	Best for text-based files
Parallel Transfers	`parallel -j4 rsync -avz {}`	Best for large datasets
Resume Transfers	`rsync --partial --append`	Best for large single files
Bandwidth Limiting	`rsync --bwlimit=5000`	Prevents network congestion
Pre-scanning Files	`find /data/ -type f -print0	rsync --files-from=-`