A little history on tar

The tar command is a command that appeared in the early days of Unix and has had several changes made over time. Originally the command was used to take files, combine them into one file and write them to a tape archive (tar). Nowadays tar is used mostly as a general purpose tool to package and compress many files into one single file for distribution or backup.

There are several common implementations of tar that are in use today, because there are multiple implementations there are also some differences in the options and formats available. In today’s article I will not be showing all of the various options of tar (that’s what man pages are for), but rather will be showing commonly used flags and some not so common tricks.

Tar Basics

Creating a tar file

To create a basic tar you really only need to specify a few things.

• -c Stands for create, you will see this a lot in our examples today
• -f or –file immediately followed by a file or device will tell tar where to create the tar file
• And finally the files or directories to package

$ tar -cf tarfile.tar file1.txt

 Extracting a tar file

Extracting a tar file is just as simple as creating one.

• -x Stands for extract
• -f or –file immediately followed by a file or device has the same usage as create

$ tar -xf tarfile.tar

Adding verbosity

By default tar does not output what it is doing, you can add this by adding verbosity to the command with the -v flag. In addition to adding verbosity we are also going to tar more than one file in our example. Packaging more than one file is the point of tar after all isn’t it?

$ tar -cvf tarfile.tar files_dir/ file1.txt
files_dir/
files_dir/file3.txt
files_dir/file4.txt
file1.txt

As you can see packaging an entire directory is as simple as adding it to the list of files to package into a tar file.

Listing files in an existing tar

Sometimes you simply want to look at the files within a tar file without extracting, to do so we can use the -t or –list flag. As a side note it is generally a good practice when you receive a tar file from an outside source to list the contents of the tarball to ensure you are not overwriting files you do not intend to.

$ tar -tf tarfile.tar
files_dir/
files_dir/file3.txt
files_dir/file4.txt
file1.txt

We can also add the verbose option and show the files attributes such as permissions, size and timestamps.

$ tar -tvf tarfile.tar
drwxrwxr-x madflojo/madflojo 0 2013-05-22 21:00 files_dir/
-rw-rw-r-- madflojo/madflojo 0 2013-05-22 21:00 files_dir/file3.txt
-rw-rw-r-- madflojo/madflojo 0 2013-05-22 21:00 files_dir/file4.txt
-rw-rw-r-- madflojo/madflojo 0 2013-05-22 20:42 file1.txt

An important note on tar is that it has the ability of retaining file attributes such as permissions, size and timestamps. When extracted as a user with proper privileges these attributes will be applied to the newly created files or overwritten files.

Appending files to an existing tar

Once a tar file is created it is possible to add files with the -r or –append option. The append option however is not allowed when the file had been compressed.

$ tar -rvf tarfile.tar file2.txt
 file2.txt

Adding gzip compression

Early versions of tar used Unix compress for file compression, after some time gzip compression was also added.

The old way

Some systems had implemented the gzip command but not a tar command that added gzip inherently. Originally if users wanted to create a tarball that was gzip compressed they would need to tar the file and then gzip it.

$ tar -cvf tarfile.tar file1.txt file2.txt
file1.txt
file2.txt
$ gzip tarfile.tar
$ ls -la tarfile.tar.gz
-rw-rw-r-- 1 madflojo madflojo 136 May 22 21:22 tarfile.tar.gz

The new way

Modern implementations of tar add gzip compression inherently; you can add this compression at the creation of the tar file with the -z or –gzip option.

$ tar -cvzf tarfile.tar.gz file1.txt file2.txt
file1.txt
file2.txt

 Adding bzip2 compression

bzip2 is a compression tool much like gzip however it uses a different algorithm to compress files and is generally better at compression however it takes longer to compress items. To add bzip2 compression we simply add a -j to the command.

$ tar -cjvf tarfile.tar.bz files_dir
files_dir/
files_dir/file3.txt
files_dir/file4.txt

Extracting tarballs with compression

Any-time you are dealing with tarfiles that have been compressed you will need to add the appropriate compression flag to other tar commands such as extract or list. The following is an example of extracting a bzip2 file.

$ tar -xjvf tarfile.tar.bz files_dir
files_dir/
files_dir/file3.txt
files_dir/file4.txt

Listing tarballs with compression

The following is an example of listing a tar files contents that has gzip compression.

$ tar -cjvf tarfile.tar.bz files_dir
files_dir/
files_dir/file3.txt
files_dir/file4.txt

Extract without replacing old files

The tar commands on today’s systems have the ability to extract files without overwriting and existing file. To enable this you will need to specify -k on the extract command.

$ tar -czf tarfile.tar.gz file1.txt file2.txt
$ rm file2.txt && echo "I removed file2" >> file1.txt
$ tar -xvzkf tarfile.tar.gz
file1.txt
file2.txt
$ cat file1.txt
I removed file2

Beyond the basic tar commands

Creating a tar with –files-from  to avoid argument list too long

Sometimes specifying the files for tar to package is difficult. Either due to the number of files, the names of files or simply because it is too much to type. Tar has the ability to read a file and create a tarball of the files listed within the input file.

Below is an example of one way to get around the argument list too long problem.

 The problem:

$ tar -czf ../tarfile.tgz *
bash: /bin/tar: Argument list too long

Solution:

$ ls > ../filestocopy.txt
$ tar -T ../filestocopy.txt -czf ../tarfile.tgz

In addition to the argument list too long scenario the -T flag can be useful for automated jobs that may need to run tar against many files.

Tarpipe (or TarCopy)

Tarpipe or sometimes refereed to as tarcopy is the process where one would use tar to copy files from one place to another.

The idea behind tarpipe is that tar has the ability to send the packaged files to stdout rather than to a file. When you use this you can pipe that stdout to another tar command in a different directory.

$ tar -cf - file* | (cd ../files_copied/ && tar -xf -)

The – after -f where a file name would normally go is what tells tar to send the output to standard out.

Why use tar and not cp?

Originally the cp command did not support preserving timestamps and file permissions and that was one of the major reasons to use tarpipe rather than cp. However times have changed and modern-day cp commands do have the -p (preserve) option, but there is still one reason to use tarpipe over cp. It’s Faster!

$ time tar -cf - file* | (cd ../files_copied/ && tar -xf -)

real    0m0.010s
user    0m0.004s
sys    0m0.004s

$ time cp -p file* ../files_copied/

real    0m0.024s
user    0m0.000s
sys    0m0.000s

While .006s does not seem like a long time the above command only copied 2 files. If these files are large in size or if we start talking about millions of files, that .006s starts adding up.

Using tarpipe to copy files to a remote system

Sometimes you may need to copy files from one system to another retaining permissions and timestamps. Luckily tarpipe isn’t only limited to local system copies, you can also use it to copy to remote systems through SSH. While on most modern systems its probably better/faster to use rsync, if you are supporting an older OS that doesn’t have rsync this could save you sometime.

$ tar -cf - file* | ssh remote-server "(cd /files_copied/ && tar -xf -)"

A question came up on the Stack Exchange site Unix & Linux in which I wrote up a pretty good answer, that describes some of the mechanics of how a process deals with its user credentials, so I’m adding my writeup to the blog.

It really comes down to what makes up a process in Unix. A process can come into existence in one of 2 ways. Either via the fork() function or through one of the exec() functions in C.

fork()

fork() basically just makes a copy of the current process, but assigns it a new process ID (PID). It’s a child of the original process. You can see this relationship in the output of @ps@:

1
2
3
4
5
6
7

$ ps axjf
 PPID   PID  PGID   SID TTY      TPGID STAT   UID   TIME COMMAND
    1  5255  1964  1964 ?           -1 Sl     500   0:39 gnome-terminal
 5255  5259  1964  1964 ?           -1 S      500   0:00  \_ gnome-pty-helper
 5255 18422 18422 18422 pts/1    18422 Ss+    500   0:01  \_ bash
 5255 30473 30473 30473 pts/4    30473 Ss+    500   0:00  \_ bash
30473   782   782 30473 pts/4    30473 Sl     500   1:14  |   \_ evince s.pdf

Here you can see that gnome-terminal is the parent process (PID = 5255) and that bash is it’s child (PID = 18422, PPID = 5255).

When a process forks from its parent, it “inherits” certain things, such as copies of all the file descriptors that the parent currently has for open files and the parent’s user and group IDs.

NOTE1: PPID = Parent Process ID.
NOTE2: The last 2 are what identify what file and group permissions this process will have when accessing the file system.

So if a process just inherits its user and group ID from its parent, then why isn’t everything just owned by root or a single user? This is where exec() comes in.

exec() Part #1

The exec() family of functions, specifically execve(), “replace” a current process image with a new process image. The terminology “process image” is really just a file, i.e. an executable on disk. So this is how a bash script can execute a program such as /usr/bin/time.

So what about the user ID and group ID? Well to understand that let’s first discuss the concept of “Persona”.

Persona

At any time, each process has an effective user ID, an effective group ID, and a set of supplementary group IDs. These IDs determine the privileges of the process. They are collectively called the [persona of the process]¹, because they determine “who it is” for purposes of access control.

exec() Part #2

So in addition to being able to swap out the “process image”, exec() can also change the user & group IDs from the original “real” ones to “effective” ones.

An example

For this demonstration I’m going to show you what happens when we start out in a shell as our default UID/GID, and then spawn a child shell using one of my supplementary GIDs, making it the child shell’s effective GID.

To perform this I’m going to make use of the unix command newgrp. newgrp allows you to spawn a new shell passing it the supplementary group that I’d like to make my effective GID.

For starters:

1
2

$ id -a
uid=500(saml) gid=501(saml) groups=501(saml),502(vboxusers),503(jupiter)

We can see that this shell is currently configured with my default UID/GID of saml & saml. Touching some files shows that this is the case as well:

1
2
3
4
5
6

$ touch afile1
$ touch afile2
$ ls -l
total 0
-rw-rw-r-- 1 saml saml 0 May 21 23:47 afile1
-rw-rw-r-- 1 saml saml 0 May 21 23:47 afile2

Now we make our supplementary group jupiter the effective GID:

1
2
3

$ newgrp jupiter
$ id -a
uid=500(saml) gid=503(jupiter) groups=501(saml),502(vboxusers),503(jupiter)

Now if we touch some files:

1
2
3
4
5
6
7
8

$ touch afile3
$ touch afile4
$ ls -l
total 0
-rw-rw-r-- 1 saml saml    0 May 21 23:47 afile1
-rw-rw-r-- 1 saml saml    0 May 21 23:47 afile2
-rw-r--r-- 1 saml jupiter 0 May 21 23:49 afile3
-rw-r--r-- 1 saml jupiter 0 May 21 23:49 afile4

We see that the shell’s effective GID is jupiter, so any interactions with the disk result in files being created with jupiter rather than my normal default group of saml.

References

• fork() man page
• exec() man page
• execve() man page
• credentials man page
• newgrp man page
• The GNU C Library
• The GNU C Library – 26.4 Creating a Process
• The GNU C Library – 26.5 Executing a File
• GID, current, primary, supplementary, effective and real group IDs?

rdiff-backup is a rsync delta based backup tool that both stores a full mirror and incremental changes. It determines changes based on the rsync method of creating small delta files, which allows for rdiff-backup to restore files to any point in time (within the specified retention period).

In the examples below I will refer to two servers names, backup-server and server. The names are pretty self-explanatory but just in case, backup-server is the location where I permanently store files copied (backed up) from server.

Setting up rdiff-backup

Installing rdiff-backup is easy considering most Linux distributions include it into their default repositories. In this article I will be using Ubuntu for my example systems.

Note: For Red Hat you will need to enable the EPEL repository to install rdiff-backup via YUM.

Installing

In order for rdiff-backup to work both the source and destination will require the rdiff-backup package. You can install it via apt-get.

On backup-server:

root@backup-server# apt-get install rdiff-backup

On server:

root@server# apt-get install rdiff-backup

Validate rdiff-backup versions match

One of the quirky things about rdiff-backup is that the tool does not support backwards capability with older versions. For this reason it is best to make sure that your rdiff-backup versions are the same on both servers.

On backup-server:

root@backup-server# rdiff-backup --version
rdiff-backup 1.2.8

On server:

root@server# rdiff-backup --version
rdiff-backup 1.2.8

Setting up SSH Keys

By default rdiff-backup uses SSH to communicate with remote systems to avoid typing a password every time rdiff-backup runs we will need to set-up SSH keys with passphrase-less authentication.

On backup-server:

root@backup-server# ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.

When asked leave the passphrase empty.

Once you have the SSH key generated you will need to copy the contents of /root/.ssh/id_rsa.pub to the remote servers for key-based authentication. For our configuration we will use a non-privileged user account (test), as this will let us implement rdiff-backup without giving the backup-server full access to the systems being backed up.

On backup-server:

root@backup-server:# scp /root/.ssh/id_rsa.pub test@server:/var/tmp/id_rsa.pub.temp

On server:

test@server:$ cat /var/tmp/id_rsa.pub.temp >> ~/.ssh/authorized_keys

You should now be able to SSH from backup-server to server without being asked for a password.

Running backup jobs

Now that backup-server is able to SSH to server without being asked a password and rdiff-backup is the same version on both systems we are able to perform the first backup.

The directory we will backup today is /var/tmp/backmeup and we will be backing it up to /var/tmp/backups/server.example.com/. I personally prefer to backup to a directory named after the originating server, that way there is no question as to where the files came from.

On backup-server:

root@backup-server:# mkdir -p /var/tmp/backups/server.example.com
root@backup-server:# rdiff-backup test@server.example.com::/var/tmp/backmeup /var/tmp/backups/server.example.com/

rdiff-backup has now created a mirror of the /var/tmp/backmeup directory from server.example.com in /var/tmp/backups/server.example.com.

root@backup-server:# ls -la /var/tmp/backups/server.example.com/
total 52
drwxr-xr-x 3 root root 4096 May 19 13:07 .
drwxr-xr-x 3 root root 4096 May 19 13:53 ..
-rw-r--r-- 1 root root   25 May 19 13:07 10.file
-rw-r--r-- 1 root root   24 May 19 13:07 1.file
-rw-r--r-- 1 root root   24 May 19 13:07 2.file
-rw-r--r-- 1 root root   24 May 19 13:07 3.file
-rw-r--r-- 1 root root   24 May 19 13:07 4.file
-rw-r--r-- 1 root root   24 May 19 13:07 5.file
-rw-r--r-- 1 root root   24 May 19 13:07 6.file
-rw-r--r-- 1 root root   24 May 19 13:07 7.file
-rw-r--r-- 1 root root   24 May 19 13:07 8.file
-rw-r--r-- 1 root root   24 May 19 13:07 9.file
drwx------ 3 root root 4096 May 19 13:56 rdiff-backup-data

Now that we have backed up the original file we will run a second backup to capture changed data; this time a with a little more verbosity.

root@backup-server:# rdiff-backup -v5 test@server.example.com::/var/tmp/backmeup /var/tmp/backups/server.example.com/
Using rdiff-backup version 1.2.8
Executing ssh -C test@server.example.com rdiff-backup --server
<truncated for length>
Backup: must_escape_dos_devices = 0
Starting increment operation /var/tmp/backmeup to /var/tmp/backups/server.example.com
Processing changed file .
Incrementing mirror file /var/tmp/backups/server.example.com
Processing changed file 1.file
Incrementing mirror file /var/tmp/backups/server.example.com/1.file
Processing changed file 10.file
Incrementing mirror file /var/tmp/backups/server.example.com/10.file
Processing changed file 2.file
Incrementing mirror file /var/tmp/backups/server.example.com/2.file
Processing changed file 3.file
Incrementing mirror file /var/tmp/backups/server.example.com/3.file
Processing changed file 4.file
Incrementing mirror file /var/tmp/backups/server.example.com/4.file
Processing changed file 5.file
Incrementing mirror file /var/tmp/backups/server.example.com/5.file
Processing changed file 6.file
Incrementing mirror file /var/tmp/backups/server.example.com/6.file
Processing changed file 7.file
Incrementing mirror file /var/tmp/backups/server.example.com/7.file
Processing changed file 8.file
Incrementing mirror file /var/tmp/backups/server.example.com/8.file
Processing changed file 9.file
Incrementing mirror file /var/tmp/backups/server.example.com/9.file

As you can see -v5 tells us what files are being processed, this is handy to see what is being backed up or being restored.

Now if we only change files 1 – 3 and run rdiff-backup again rdiff-backup should only backup files that have changed leaving the others alone.

root@backup-server:# rdiff-backup -v5 test@server.example.com::/var/tmp/backmeup /var/tmp/backups/server.example.com/
Using rdiff-backup version 1.2.8
Executing ssh -C test@server.example.com rdiff-backup --server
<truncated for length>
Starting increment operation /var/tmp/backmeup to /var/tmp/backups/server.example.com
Processing changed file .
Incrementing mirror file /var/tmp/backups/server.example.com
Processing changed file 1.file
Incrementing mirror file /var/tmp/backups/server.example.com/1.file
Processing changed file 2.file
Incrementing mirror file /var/tmp/backups/server.example.com/2.file
Processing changed file 3.file
Incrementing mirror file /var/tmp/backups/server.example.com/3.file

If we look at the backup directory the number of files has not changed, however the contents and time stamps have.

root@backup-server:# ls -la /var/tmp/backups/server.example.com/
total 52
drwxr-xr-x 3 root root 4096 May 19 13:07 .
drwxr-xr-x 3 root root 4096 May 19 13:53 ..
-rw-r--r-- 1 root root   76 May 19 14:10 10.file
-rw-r--r-- 1 root root   98 May 19 14:16 1.file
-rw-r--r-- 1 root root   98 May 19 14:16 2.file
-rw-r--r-- 1 root root   98 May 19 14:16 3.file
-rw-r--r-- 1 root root   73 May 19 14:10 4.file
-rw-r--r-- 1 root root   73 May 19 14:10 5.file
-rw-r--r-- 1 root root   73 May 19 14:10 6.file
-rw-r--r-- 1 root root   73 May 19 14:10 7.file
-rw-r--r-- 1 root root   73 May 19 14:10 8.file
-rw-r--r-- 1 root root   73 May 19 14:10 9.file
drwx------ 3 root root 4096 May 19 14:16 rdiff-backup-data

rdiff-backup will keep the current mirror unchanged and any differences will be kept in diff files within the rdiff-backup-data directory. It is not advised to modify or interact with the mirror or diff files directly, it is better to use the rdiff-backup command itself.

Listing available backups

To see the available backups we can use rdiff-backup -l.

root@backup-server:# rdiff-backup -l /var/tmp/backups/server.example.com/
Found 5 increments:
    increments.2013-05-19T13:56:57-07:00.dir   Sun May 19 13:56:57 2013
    increments.2013-05-19T14:09:52-07:00.dir   Sun May 19 14:09:52 2013
    increments.2013-05-19T14:11:29-07:00.dir   Sun May 19 14:11:29 2013
    increments.2013-05-19T14:16:44-07:00.dir   Sun May 19 14:16:44 2013
    increments.2013-05-19T14:29:38-07:00.dir   Sun May 19 14:29:38 2013
Current mirror: Sun May 19 14:30:20 2013

If a file has been deleted and rdiff-backup has ran since the file deletion you may not find the file in the directory, you can still however list the available backups for that file by specifying it as if it did exist.

 root@backup-server:# rdiff-backup -l /var/tmp/backups/server.example.com/1.file
Found 4 increments:
    1.file.2013-05-19T13:56:57-07:00.diff.gz   Sun May 19 13:56:57 2013
    1.file.2013-05-19T14:09:52-07:00.diff.gz   Sun May 19 14:09:52 2013
    1.file.2013-05-19T14:11:29-07:00.diff.gz   Sun May 19 14:11:29 2013
    1.file.2013-05-19T14:16:44-07:00.snapshot.gz   Sun May 19 14:16:44 2013
Current mirror: Sun May 19 14:30:20 2013

Restoring backed up files and directories

rdiff-backup has the ability to restore either individual files or entire directories, as long as rdiff-backup has the item within its incremental lists.

Restoring an individual file

When restoring an individual file with rdiff-backup you can either specify a time or the incremental file to restore from. For  the following example I will show using the incremental file.

root@backup-server:# cd server.example.com/rdiff-backup-data/increments/
root@backup-server:# rdiff-backup -v5 1.file.2013-05-19T14\:11\:29-07\:00.diff.gz test@server.example.com::/var/tmp/backmeup/1.file

Restoring a directory

When restoring a directory however we will need to specify a specific time that we want to restore to.

root@backup-server:# rdiff-backup -v5 -r 1h server.example.com/ test@server.example.com::/var/tmp/backmeup

This command will restore the entire directory to where it was 1 hour ago or best it can depending on the backups available. rdiff-backup can support many time frames but I commonly find myself using the xDays format (e.g. 2D for 2 days).

Don’t use the force flag

While the above command will restore the whole directory it will only do so if the directory is empty. If the directory has files in it and you ask rdiff-backup to restore that directory than it will try to remove the existing files in order to match your backup. This action could result in data that has not been backed up being removed.

To protect against accidental deletion rdiff-backup requires the force flag to be used anytime a file is being overwritten or deleted.

root@backup-server:# rdiff-backup -v5 -r 1h server.example.com/ test@server.example.com::/var/tmp/backmeup
Using rdiff-backup version 1.2.8
Executing ssh -C server.example.com rdiff-backup --server
Fatal Error: Restore target /var/tmp/backmeup already exists, specify --force to overwrite.

I advise avoiding the use of the force flag whenever possible, if you truly do not want the contents of the directory than just remove them manually before restoring. I have seen many times where people used the force flag and accidentally overwrote a directory they did not mean (like /etc/ for example…).

Restoring to another location

When restoring with rdiff-backup you can restore files or directories to a location other than their originating source. This can be handy if you need to check the contents before completely restoring the file.

root@backup-server:# rdiff-backup -v5 -r 3h server.example.com/1.file test@server.example.com::/var/tmp/backmeup/1.file.restore

Backup Retention

Backups are only as good as their retention period, without a retention period you will eventually run out of disk space or use far more disk space than you had originally planned. rdiff-backup has the ability to maintain a certain number of incremental copies. With rdiff-backup you can tell it to either keep a backup for a certain amount of time or for a certain number of backups.

On backup-server:

Time method

The time method uses the same time format as restore.

root@backup-server:# rdiff-backup --force --remove-older-than 4h /var/tmp/backups/server.example.com

Number of backups method

To specify a number of backups use the number followed by a capital B.

root@backup-server:# rdiff-backup --force --remove-older-than 4B /var/tmp/backups/server.example.com

I used the force flag with the above commands as rdiff-backup requires force to be given if you are removing more than one incremental copy.

Providing more access with sudo

So far we have been backing up files and directories that the test user has access to; if we were to try and backup or restore a file that the test user does not have access to than the backup/restore will fail with a permission denied. To provide greater access you can either run rdiff-backup as the root user on the remote systems (which raises security concerns), or provide the test user with the ability to run rdiff-backup as the root user via sudo.

Example of permission denied error:

root@backup-server:# rdiff-backup -v5 test@server.example.com::/var/tmp/backmeup /var/tmp/backups/server.example.com
Using rdiff-backup version 1.2.8
Executing ssh -C test@server.example.com rdiff-backup --server
Exception '[Errno 13] Permission denied: '/var/tmp/backmeup'' raised of class '<type 'exceptions.OSError'>':

Adding the rdiff-backup into /etc/sudoers

In order to allow the test user the ability to run rdiff-backup as root we need to add an entry into the /etc/sudoers file, which controls what commands users can run via sudo. To modify this file we will use the visudo command.

On server:

root@server:/var/tmp# visudo

Append:

## Give test user the ability to run rdiff-backup
test    ALL = NOPASSWD: /usr/bin/rdiff-backup --server

As the test user you will now see rdiff-backup in the list of available sudo commands

test@server:~$ sudo -l
User test may run the following commands on this host:
    (root) NOPASSWD: /usr/bin/rdiff-backup --server

We are specifying NOPASSWD as by default sudo would normally ask the user for their password, which would not work very well with an automated backup script.

Running rdiff-backup with remote-schema

In order for rdiff-backup to use sudo we will need to change the command we have been using a bit; we will use the –remote-schema flag to tell rdiff-backup to run “sudo /usr/bin/rdiff-backup –server” on the remote system.

On backup-server:

Backup command

root@backup-server:# rdiff-backup -v5 --remote-schema 'ssh -C %s "sudo /usr/bin/rdiff-backup --server"' \
test@server.example.com::/var/tmp/backmeup /var/tmp/backups/server.example.com

<truncated>
Processing changed file 9.file
Incrementing mirror file /var/tmp/backups/server.example.com/9.file

Restore command

root@backup-server:# rdiff-backup -v5 -r 3h --remote-schema 'ssh -C %s "sudo /usr/bin/rdiff-backup --server"' \
/var/tmp/backups/server.example.com/5.file test@server.example.com::/var/tmp/backmeup/5.file

By adding sudo we are allowing the test user to backup and restore any file on the system with rdiff-backup.

Adding restrict-read-only for even more security

While using rdiff-backup with sudo prevents people from using the SSH key to login as root to all of our remote systems. This solution by itself does not restrict someone from using rdiff-backups restore function from deploying compromised files.

For even more security we can use the –restrict-read-only flag to restrict rdiff-backup to only being able to read files and blocking all write requests. The down side of this setting is that it also prevents valid restore requests as well. If you are more worried about someone accessing your systems than having to edit the sudoers file every time you want to restore a file; than this is a good option.

Adding restrict-read-only to the sudoers entry

In order to add –restrict-read-only we need to add it to both the rdiff-backup command and the sudoers entry.

root@server# visudo

Modify to:

test    ALL = NOPASSWD: /usr/bin/rdiff-backup --server --restrict-read-only /

The / at the end is the path that you want rdiff-backup to be restricted to. This entry would give rdiff-backup the ability to backup all files on the system. If you are not backing up the entire system you can restrict this to a specific path as well to prevent rdiff-backup from reading other files on the system not within your path.

Running the backup command with restrict-read-only

Now that sudo allows us to run the full command we can add it to the remote-schema.

root@backup-server:# rdiff-backup -v5 --remote-schema 'ssh -C %s "sudo /usr/bin/rdiff-backup --server --restrict-read-only /"' \
 test@server.example.com::/var/tmp/backmeup /var/tmp/backups/server.example.com
Using rdiff-backup version 1.2.8
Executing ssh -C test@server.example.com "sudo /usr/bin/rdiff-backup --server"

If you modified the path in the sudoers file you would need to do the same with the rdiff-backup command above.

Automating with Cron

Automating rdiff-backup with cron is as simple as tossing the commands above into a script and adding it to the crontab. The below is meant only for example, I would advise anyone reading this to script in some more intelligence to handle failed backups and concurrent runs but if you needed something quick and dirty this would work.

On backup-server:

Creating the backup script

root@backup-server# vi /root/backup-example.sh

Add:

#!/bin/bash
## Example rdiff-backup script - http://bencane.com
## This is not fancy, and you should really add error checking

# Backup
rdiff-backup -v5 --remote-schema 'ssh -C %s "sudo /usr/bin/rdiff-backup --server --restrict-read-only /"' \
 test@server.example.com::/var/tmp/backmeup /var/tmp/backups/server.example.com

# Clean Increments
rdiff-backup --force --remove-older-than 4B /var/tmp/backups/server.example.com

Adding to crontab

Once you have the script you can simply add the script into the crontab on the backup-server.

root@backup-server# crontab -e

Append:

# m h  dom mon dow   command
0 0 * * * /root/backup-example.sh > /dev/null 2>&1

The above crontab entry will run backup-example.sh every night at midnight. This will provide you with 4 days of incremental copies at all times.

People that use Linux on a daily basis probably are completely oblivious to the actual mechanisms being used to store their passwords safely and securely on a given Linux system. Oh they might guess that their password is stored in the /etc/passwd file (they’d be wrong by the way) but most probably never even gave it a passing thought. So I thought I’d take the opportunity to shed some light on how Linux systems “stash” your precious password away.

Solution

So if your password isn’t actually stored in the /etc/passwd file then where does it get stored?

Answer: the /etc/shadow file.

This file is where all the keys to each user’s account are kept for safe keeping. Obviously only the root user can peer inside this file so all the commands we’ll be dealing with in this post, it should be assumed that you’ll need to either be root, or use sudo to run.

/etc/shadow

A typical /etc/shadow entry:

1

root:$6$bbmDJwcZHy5bgEDz$kFO.W/T7nUqcszZWl5RglxoDDAcDxevWpHVfN3v3f.Cx2ZeMcn5PX23VvnnkgtNWZf8hYtqsL0pPkZqyj50NY/:14362:0:33333:7:::

NOTE1: Don’t get too excited, the above isn’t really my entry, I made this one up.
NOTE2: Each field is separated by a colon (:) & we’re only concerned with the first two columns!

dissecting the hash

The key pieces to notice in that line of what looks like gibberish is the following:

• The first column, root is the user whom this entry belongs to from the /etc/passwd file.
• The second column, $6$..... is essentially the user’s hashed password.

Taking the second column apart further you should start to notice that’s it’s not complete gibberish after all.

For example:

• the first couple of characters, $6$, is a mark that tells the system what type of hashing was used to hash the password.
• The text between the next set of dollar signs, $bbmDJwcZHy5bgEDz$, is the actual salt that was used to hash your password.
• Everything else after, is your password + salt hashed using whatever hash function was specified at the beginning, $6$, in our example here.

Specifically if you look at the man page for the crypt command, man 3 crypt there is a section that discusses what the $6$ notation means:

So $5$salt$encrypted is an SHA-256 encoded password and $6$salt$encrypted is an SHA-512 encoded one.

NOTE: So in our case the password + salt is being hashed using the SHA-512 scheme.

design details

For reference purposes here’s the rest of that excerpt from the crypt man page:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31

If salt is a character string starting with the characters "$id$" followed by a
string terminated by "$":
 
       $id$salt$encrypted
 
then instead of using the DES machine, id identifies the encryption method used
and this then determines how the rest of the password string is interpreted.
The following values of id are supported:
 
       ID  | Method
       ─────────────────────────────────────────────────────────
       1   | MD5
       2a  | Blowfish (not in mainline glibc; added in some
           | Linux distributions)
       5   | SHA-256 (since glibc 2.7)
       6   | SHA-512 (since glibc 2.7)
 
So $5$salt$encrypted is an SHA-256 encoded password and $6$salt$encrypted is an
SHA-512 encoded one.
 
"salt" stands for the up to 16 characters following "$id$" in the salt. The
encrypted part of the password string is  the actual computed password. The
size of this string is fixed:
 
MD5     | 22 characters
SHA-256 | 43 characters
SHA-512 | 86 characters
 
The characters in "salt" and "encrypted" are drawn from the set [a–zA–Z0–9./].
In the MD5 and SHA implementations the entire key is significant (instead of
only the first 8 bytes in DES).

Now what?

So by now you’re probably saying to yourself. OK, big deal, my password is hashed with some salt and stored in /etc/shadow. What else?

generating the hash manually using mkpasswd

For starters you can generate the $6$... string yourself manually using the mkpasswd command:

1
2

$ mkpasswd -m sha-512 password saltsalt
$6$saltsalt$qFmFH.bQmmtXzyBY0s9v7Oicd2z4XSIecDzlB5KiA2/jctKu9YterLp8wwnSq.qc.eoxqOmSuNp2xS0ktL3nh/

In the above command we’re specifying that we want to use the SHA-512 hash, our password is the string password and our salt string is saltsalt. As before we can see in our resulting string the following components:

• $6$ – which hash function was used
• saltsalt – the string “saltsalt” was used
• qFmFH.bQmmtXzyBY0s9v7Oicd2z4XSIecDzlB5KiA2/jctKu9YterLp8wwnSq.qc.eoxqOmSuNp2xS0ktL3nh/ – password + salt hashed using SHA-512

generating the hash manually using Python

I came across the following nice Python one-liner that effectively does the same thing as the mkpasswd command discussed above.

1
2
3

$ python -c "import crypt, getpass, pwd; \
             print crypt.crypt('password', '\$6\$saltsalt\$')"
$6$saltsalt$qFmFH.bQmmtXzyBY0s9v7Oicd2z4XSIecDzlB5KiA2/jctKu9YterLp8wwnSq.qc.eoxqOmSuNp2xS0ktL3nh/

generating the hash manually using Perl

1
2

$ perl -e 'print crypt("password","\$6\$saltsalt\$") . "\n"'
$6$saltsalt$qFmFH.bQmmtXzyBY0s9v7Oicd2z4XSIecDzlB5KiA2/jctKu9YterLp8wwnSq.qc.eoxqOmSuNp2xS0ktL3nh/

authconfig

Before I wrap up I thought I’d mention one final tool authconfig that’s included on Red Hat distros such as Fedora, CentOS, and RHEL. This tool allows you to change what hash algorithm is being used on a particular system. The command to change a system to use SHA-512 would be as follows:

1

authconfig –passalgo sha512 –update

See the man page for authconfig for more details.

conclusions

And with that you are now a little more in the know as to how Linux systems take your password and store them in the /etc/shadow file.

References

links

• crypt man page
• Thread: generate hashed password for /etc/shadow – ubuntuforums.org
• How to produce test hashes for various formats
• Shadow Fu
• Using ruby to generate SHA512 crypt-style hashes formatted for /etc/shadow?
• Calculate SHA Hash of Linux Password in Shadow File

Problem

I’m using rsync to copy some files from a share to another.

Recursively, I need to:

- delete files at the destination that are deleted in the origin
- Only sync php and js files
- exclude de rest of file types
- Don’t delete .svn/ directory in the destination

If I use this:

rsync -zavC --delete --include='*.php' --include='*.js' --exclude="*" /media/datacod/Test/ /home/lucas/Desktop/rsync/

Then rsync is not recursive because exclude=”*” excludes all files but also folders

If I add --include="*/" then the .svn/ directory gets deleted (it also gets included)

How can I solve this mind blasting dilemma?

Solution

The solution I ultimately came up with made use of a little known feature, at least to me, called filters. Filters allow you to play games with the includes/excludes by protecting portions based on regular expressions. Read on, I’ll discuss them further down.

1
2

rsync -avzC --filter='-rs_*/.svn*' --include="*/" --include='*.js' --include='*.php' \
     --exclude="*" --delete dir1/ dir2/

test data

To help determine if my solution was going to work or not I created some sample data so that I could test it out. For starters I wrote a script that would generate the data. Here’s that script, setup_svn_sample.bash:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

#!/bin/bash
 
# setup .svn dirs
mkdir -p dir{1,2}/dir{1,2,3,4}/.svn
 
# fake data under .svn
mkdir -p dir1/dir{1,2,3,4}/.svn/origdir
mkdir -p dir2/dir{1,2,3,4}/.svn/keepdir
 
# files to not sync
touch dir1/dir{1,2,3,4}/file{1,2}
 
# files to sync
touch dir1/dir{1,2,3,4}/file1.js
touch dir1/dir{1,2,3,4}/file1.php

Running the above script produces the following directories (dir1 & dir2):

source dir

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

$ tree -a dir1
dir1
|-- dir1
|   |-- file1
|   |-- file1.js
|   |-- file1.php
|   |-- file2
|   `-- .svn
|       `-- origdir
|-- dir2
|   |-- file1
|   |-- file1.js
|   |-- file1.php
|   |-- file2
|   `-- .svn
|       `-- origdir
|-- dir3
|   |-- file1
|   |-- file1.js
|   |-- file1.php
|   |-- file2
|   `-- .svn
|       `-- origdir
`-- dir4
    |-- file1
    |-- file1.js
    |-- file1.php
    |-- file2
    `-- .svn
        `-- origdir

destination dir

1
2
3
4
5
6
7
8
9
10
11
12
13
14

$ tree -a dir2
dir2
|-- dir1
|   `-- .svn
|       `-- keepdir
|-- dir2
|   `-- .svn
|       `-- keepdir
|-- dir3
|   `-- .svn
|       `-- keepdir
`-- dir4
    `-- .svn
        `-- keepdir

Running the above rsync command which includes the --filter below we can see that it’s only syncing the files that match the --include patterns:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

rsync -avzC --filter='-rs_*/.svn*' --include="*/" --include='*.js' --include='*.php' \
     --exclude="*" --delete dir1/ dir2/
sending incremental file list
dir1/file1.js
dir1/file1.php
dir2/file1.js
dir2/file1.php
dir3/file1.js
dir3/file1.php
dir4/file1.js
dir4/file1.php
 
sent 480 bytes  received 168 bytes  1296.00 bytes/sec
total size is 0  speedup is 0.00

Resulting dir2 afterwards:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

$ tree -a dir2
dir2
|-- dir1
|   |-- file1.js
|   |-- file1.php
|   `-- .svn
|       `-- keepdir
|-- dir2
|   |-- file1.js
|   |-- file1.php
|   `-- .svn
|       `-- keepdir
|-- dir3
|   |-- file1.js
|   |-- file1.php
|   `-- .svn
|       `-- keepdir
`-- dir4
    |-- file1.js
    |-- file1.php
    `-- .svn
        `-- keepdir

Why does it work?

The key piece to this script is to make use of the filters capability of rsync. Filters allow you to remove files from the matched set at various points in the command. So in our case we’re filtering any files that match the pattern */.svn*. The modifiers -rs_ tell the filter that we want to filter on both the source side as well as the target side.

excerpt from the FILTER NOTES section of rsync’s man page

- An s is used to indicate that the rule applies to the sending side. When a rule affects the sending side, it prevents files from being
transferred. The default is for a rule to affect both sides unless --delete-excluded was specified, in which case default rules become sender-side only. See also the hide (H) and show (S) rules, which are an alternate way to specify sending-side includes/excludes.

- An r is used to indicate that the rule applies to the receiving side. When a rule affects the receiving side, it prevents files from being deleted. See the s modifier for more info. See also the protect (P) and risk ® rules, which are an alternate way to specify receiver-side includes/excludes.

See man rsync for more details.

Tips for figuring this out (hint using --dry-run)

While describing how to do this I thought I’d mention the --dry-run switch to rsync. It’ extremely useful in seeing what will happen without having the rsync actually take place.

For Example

Using the following command will do a test run and show us the decision logic behind rsync:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48

rsync --dry-run -avvzC --filter='-rs_*/.svn*' --include="*/" \
     --include='*.js' --include='*.php' --exclude="*" --delete dir1/ dir2/
sending incremental file list
[sender] showing directory dir3 because of pattern */
[sender] showing directory dir2 because of pattern */
[sender] showing directory dir4 because of pattern */
[sender] showing directory dir1 because of pattern */
[sender] hiding file dir1/file1 because of pattern *
[sender] showing file dir1/file1.js because of pattern *.js
[sender] hiding file dir1/file2 because of pattern *
[sender] showing file dir1/file1.php because of pattern *.php
[sender] hiding directory dir1/.svn because of pattern */.svn*
[sender] hiding file dir2/file1 because of pattern *
[sender] showing file dir2/file1.js because of pattern *.js
[sender] hiding file dir2/file2 because of pattern *
[sender] showing file dir2/file1.php because of pattern *.php
[sender] hiding directory dir2/.svn because of pattern */.svn*
[sender] hiding file dir3/file1 because of pattern *
[sender] showing file dir3/file1.js because of pattern *.js
[sender] hiding file dir3/file2 because of pattern *
[sender] showing file dir3/file1.php because of pattern *.php
[sender] hiding directory dir3/.svn because of pattern */.svn*
[sender] hiding file dir4/file1 because of pattern *
[sender] showing file dir4/file1.js because of pattern *.js
[sender] hiding file dir4/file2 because of pattern *
[sender] showing file dir4/file1.php because of pattern *.php
[sender] hiding directory dir4/.svn because of pattern */.svn*
delta-transmission disabled for local transfer or --whole-file
[generator] risking directory dir3 because of pattern */
[generator] risking directory dir2 because of pattern */
[generator] risking directory dir4 because of pattern */
[generator] risking directory dir1 because of pattern */
[generator] protecting directory dir1/.svn because of pattern */.svn*
dir1/file1.js
dir1/file1.php
[generator] protecting directory dir2/.svn because of pattern */.svn*
dir2/file1.js
dir2/file1.php
[generator] protecting directory dir3/.svn because of pattern */.svn*
dir3/file1.js
dir3/file1.php
[generator] protecting directory dir4/.svn because of pattern */.svn*
dir4/file1.js
dir4/file1.php
total: matches=0  hash_hits=0  false_alarms=0 data=0
 
sent 231 bytes  received 55 bytes  572.00 bytes/sec
total size is 0  speedup is 0.00 (DRY RUN)

In the above output you can see that the ./svn directories are being protected by our filter rule. Valuable insight for debugging the rsync.

References

- Delete extraneous files from dest dir via rsync?
- Above scripts in a tarball

zypper ar http://ftp.gwdg.de/pub/linux/packman/suse/openSUSE_12.3 packman
zypper in gstreamer-plugins-bad-orig-addon gstreamer-plugins-libav
gstreamer-plugins-ugly-orig-addon w32codec-all
Add the Packman repository and install the required packages.

Aside:  If you don't want to be pestered about package vendor changes in the future you can edit  /etc/zypp/zypp.conf and set "solver.allowVendorChange = true".  But don't do that unless you know what that means.