rsync command in Linux

In this article, we discuss the rsync command which is used for copying and synchronizing files and directories both locally and remotely.

Table of contents.

1. Introduction.
2. Syntax.
3. Commands.
4. Summary.
5. References.

Introduction.

rsync(remote sync) is a command line utility used to synchronize files, directories, disks, between two locations over a remote shell or rsync deamon. This can also be done to local files and directories.

It is commonly used for mirroring data, copying files between systems and incremental backups both locally or remotely. Its copying functionality is similar scp, sftp, cp commands in Linux and thus rsync can be used in their place.

Syntax.

The syntax is as follows,

rsync [OPTION]... SRC [SRC]... DEST
rsync [OPTION]... SRC [SRC]... [USER@]HOST:DEST
rsync [OPTION]... SRC [SRC]... [USER@]HOST::DEST
rsync [OPTION]... SRC [SRC]... rsync://[USER@]HOST[:PORT]/DEST
rsync [OPTION]... [USER@]HOST:SRC [DEST]
rsync [OPTION]... [USER@]HOST::SRC [DEST]
rsync [OPTION]... rsync://[USER@]HOST[:PORT]/SRC [DEST]

Commonly used options include,

• -z, to compress data.
• -a, to allow recursive copying of data and preserve file attributes such as symbolic links, file permissions, ownership, timestamps(modification, access).
• -r, to copy data recursively, in this case file attributes are not preserved.
• -h, to display numbers in human readable format.
• -v, to display information while running the operation.
• -q, supress output while running the operation.
• --progress, to display transfer progress while operating.
• --delete, to remove extraneous files from destination directories if they are not in source directory.
• --remove-source-files, remove source files after synchronization/copying.
• -e, instructs rsync to use ssh protocol for remote transfers.
• -n, --dry-run, to perform a dry run, usually for testing before the actual synchronization.
• -b, to perform a backup during synchronization.

Commands.

• We can first perform a dry run to see if it works as intended using -n option as follows,

rsync -vn ~/Desktop/images/image.png ~/DIR/images

• And now to perform an actual copy,

rsync -v ~/Desktop/images/image.png ~/DIR/images

~/Desktop/images/image.png is the source directory while the latter is the destination directory.
The command will work assuming ~/Downloads/images exists, if not we add a / at the end i.e ~/Downloads/images/ and the directory will be automatically created.

• To copy multiple files locally, we write,

rsync -v ~/Documents/file1.txt ~/Documents/file2.txt DIR/

• To copy a directory and all its subdirectory and files we use the -r or -a option,

rsync -vr ~/Documents/SRCDIR ~/Documents/DESTDIR/

or

rsync -va ~/Documents/SRCDIR ~/Documents/DESTDIR/

• To copy a file or directory from a local to a remote machine we write,

rsync -av ~/Documents/SRCDIR/image.png 192.168.100.237:~/Downloads/images

Where 192.168.100.237:~/Downloads/images is the destination ip address followed by the destination directory we want to copy the image.png file to.
After you execute the command, you will be prompted for a password and after authentication the files will be copied.

• To copy multiple files or directories we write,

rsync -av ~/Documents/SRCDIR/image.png ~/Documents/SRCDIR/image1.png 192.168.100.237:~/Downloads/images

• To copy a file from the remote host to a local host we write,

rsync -av 192.168.100.237:~/Downloads/images/image.png ~/Documents/DIR/images

In this case we have specified the remote host as the source and the local host as the destination.

• Sending files to remote hosts over a network safely requires security, ssh is used to encrypt data and provide a secure tunnel between two hosts so they can share data, To define ssh protocol with rsync, we write,

rsync -e ssh ~/Documents/SRCDIR/image.png 192.168.100.237:~/Downloads/images

• To copy multiple files from a remote host to a local host using ssh, we write,

rsync -e 192.168.100.237:{~/Downloads/images.png, ~/Downloads/images.png, *.pdf} ~/home/DIR/

Notice how we have placed multiple directories within curly braces and separated them with a comma.
Similarly we can list directories in a similar fashion.

• To view progress we use the --progress option,

rsync -av --progress 192.168.100.237:~/Downloads/images/image.png ~/Documents/DIR/images

• When we are synchronizing files from both hosts, we may want to delete files or directories which are not in the source directory from the destination directory, we use the --delete option.

rsync -av --delete 192.168.100.237:~/Downloads/SRCDIR ~/Documents/DESTDIR

Now, any file that is in DESTDIR but not in SRCDIR will be removed and both directories will be in sync.

• Assuming we want to delete source files after we have synchronized them with the destination file, we can use the --remove-source-files option,

rsync -av --remove-source-files ~/Documents/SRCDIR 192.168.100.237:~/Downloads/DESTDIR 

In this case after synchronization of SRCDIR with DESTDIR the former will be deleted.

• We can also specify the size, that is --max-size to avoid transferring any file or directory larger than the specified or --min-size to avoid transferring any file smaller than the specified.
  An example

rsync -vr --max-size=500k ~/Documents/SRCDIR ~/Documents/DESTDIR/

The above command will transfer any file or directory less than or equal to 500KB.

rsync -vr --min-size=500k ~/Documents/SRCDIR ~/Documents/DESTDIR/

The command transfers any file larger than 500KB.

• These two can be combined to define a range, for example to transfer files or directory whose size is between 10MB and 50MB we write,

rsync -vr --min-size=10M --max-size=50M ~/Documents/SRCDIR ~/Documents/DESTDIR/

• rsync also allows us to set the bandwidth limit e.g for transferring files across a network. We use the --bwlimit option followed by the limit.
  An example

rsync -vr --bwlimit=20 ~/Documents/SRCDIR ~/Documents/DESTDIR/

The above command will transfer file using 20kbps network speed.

• While synchronizing file between two locations, we may want to view the differences between the two directories or files, for this we use the -i option,

rsync -vri ~/Documents/SRCDIR ~/Documents/DESTDIR/

• Normally, during a synchronization with a destination file or directory, rsync will overwrite the destination files and sync them with source, we can avoid this by using the -u option.

rsync -vru ~/Documents/SRCDIR ~/Documents/DESTDIR/

• While making backups we may want to track the files or directories using timestamps which will tell us when the backup was made, we can do this as follows,

rsync -vri ~/Documents/SRCDIR ~/Documents/DESTDIR/$(date +%Y-%m-%d)

We have just appended $(date +%Y-%m-%d), after the operation the detsination file will be named as follows,
DESTDIR\2022\01\03.

• We also have --include and --exclude options which tells rsync to include and exclude certain files or directories.
  An example
  To transfer files which include a .pdf extension and exclude those whose names start with an F we can write,

rsync -vri --include '*.pdf' --exclude 'F*' ~/Documents/SRCDIR ~/Documents/DESTDIR/$(date +%Y-%m-%d)

Summary.

rsync is versatile and fast as compared to scp since it uses a delta transfer algorithm which allows the transfer of only differences between to sets of files, directories, disks instead of the whole.

Initially it will copy everything but the next time it is used it copies only differences.

It uses compression and decompression hence smaller file sizes and thus consumes less bandwidth while sending and receiving data.
File transfer is secure since it uses ssh for encryption.

References.

1. man rsync for the manual page or rsync --help.