Migrating data

We compare several methods for copying/migrating large amounts of data. The aim is to preserve all file permissions and metadata, which will require root access on both source and target filesystems.

We include information about some options that aren't available on BusyBox's implementation of these tools, because you may be copying data to/from systems with other Unix tools installed, which do provide those options.

Warning: None of the methods below will copy the bootmanager (ext/syslinux, grub, etc) to a new drive; you'll have to install it there explicitly.

Bit-copying methods

If your source data occupies an entire partition (that is, nothing else resides on that partition), and your target partition is the same size or larger, and you're willing to have it use the same filesystem as the source, then one option is to bit-copy the whole source partition:

Boot from LiveCD and make sure that neither /dev/sourcepart nor /dev/targetpart are mounted.

Example:

dd bs=10M if=/dev/sourcepart of=/dev/targetpart

This also copies unoccupied blocks on the sourcepart to the targetpart. A more optimal variant is to use partimage, which copies only the used portions of the sourcepart. One limitation is that currently (v0.6.9, released July 2010, current as of Mar 2012) partimage doesn't support ext4 format.

Todo: Generally the target partition will have same partition type (e.g., Linux 0x83) as the source; but is this strictly necessary?

Once you've finished, the filesystem on targetpart may occupy less than the total space available on targetpart. You may want to use a tool like resize2fs to grow the partition.

Higher-level methods

The other options we'll discuss suppose that the target filesystem is already formatted and mounted as /target.

If you're proposing to copy volatile data from a running system, you may want to copy from a lvm snapshot, or do the copy from single-user mode, or from a LiveCD or other boot medium so that your source data is no longer in use during the copy. If you're just proposing to copy from /home or such, no such precautions are necessary.

If you're copying a complete filesystem, you'll probably only want to copy the /sys and /proc mountpoints, but not their current contents. Remember to copy /boot and /dev and /tmp, the last two of which may require special attention. Pay attention to other mountpoints (such as /mnt, /media) and RAM-based filesystems (such as /run and /lib/rc/init.d).

Todo: The preceding comments assume you know what you are doing, and only serve as reminders. Would be good to link to a fuller explanation.

Note: /dev is tricky nowadays on some systems since it's hidden by udev. Here's a possible solution:

mkdir /tmp/dev mount --move /dev /tmp/dev copy /dev to /mnt/sdb5 using one of the methods elsewhere on this page mount --move /tmp/dev /dev rmdir /tmp/dev

cp

Depending on what system you're working with, and what kind of data you're copying, you may be able to do a satisfactory local copy using just cp. But there are a number of limits here that prevent this from being a general solution.

The Gnu implementation of cp provides the following options (among others):

• -a: same as -RP -p/--preserve=mode,ownership,timestamps,links,context,xattr
• -v: verbose
• -l: make hard links instead of copying
• -s: make symlinks instead of copying (source file names must be absolute)
• --sparse=always
• -u, --update: don't copy files or symlinks over an existing file unless its mtime is older
• -x, --one-file-system: skip subdirectories on different volumes

The BusyBox version provides all of the Gnu options except --sparse and -u and -x, and may lack the context and xattr options to --preserve. It will preserve hard links: if /source/A and /source/B are hard links, /target/A and /target/B will also be hard links with each other (though not with the originals). BusyBox's cp seems to silently ignore the -v option.

The FreeBSD implementation provides all of these except -s and --sparse and -u, and the --preserve options from links on: in particular, it always copies hard links as separate files. The Mac version shares the limitations of the FreeBSD version, and additionally lacks -x. Also, on the Mac version the -a shortcut isn't available, you must explicitly say -RPp.

Note: As just explained, some of these implementations will break hard links.

Warning: Avoid the lower-case -r option. Its behavior varies between implementations, and may diverge from what you want for special files or symlinks.

tar

Options recognized by BusyBox and other implementations:

• -c: create a new archive, records existing perms and mtimes, symbolic owner/group (can specify --numeric-owner), hard links among archive elements
• -x: extract files from an archive, uses current owner/group unless root, subtracts current umask unless root or -p, when extracting hard links must include first
• -t: list the contents of an archive

• -f archive: use -f - for stdin/stdout

Note: On some but not all implementations, the archive defaults to stdin/stdout when -f is not supplied. It's most portable to always declare this explicitly.

• -O: extract files to standard output
• -C dir: change to directory dir, when creating this is order-sensitive
• -v: verbose

• -z: compress archive with gzip
• -j: compress archive with bzip2
• -Z: compress archive with compress

• -h: follow symlinks, archive and extract the files they point to (like -L in other programs)
• -m: don't extract file's original mtime, leave touched with the extraction time
• -T file: get names to extract or create from file, which can also include "-C/etc" lines
• --exclude=pattern: exclude specified glob patterns
• -X file: exclude glob patterns listed in file

Note: Default globbing rules: (i) matches after any /, not just at ^; (ii) case-sensitive; (iii) wildcards match /

The following common options are allegedly also honored by BusyBox (according to comments in its source) but aren't declared in its --help:

• -o, --no-same-owner: (BSD doesn't recognize a long option) extract as yourself (default for ordinary users)

• -p, --same-permissions: (BSD doesn't recognize this long option) extract permissions verbatim, instead of subtracting current umask (usually default for root)

• --no-same-permissions: subtract umask from extracted permissions (default for ordinary users)

• --numeric-owner: create using numbers for user/groups

• -k: don't replace existing files when extracting

• --overwrite: (not on BSD) more aggressively overwrite existing files and directory metadata when extracting
  tar's default behavior is to first remove existing files (otherwise, all their hardlinks would be modified) and symlinks. If an existing dir is nonempty, its existing contents won't be removed; tar will instead just update the dir's metadata. --overwrite instead follows existing symlinks, and removes anything that impedes extraction except non-empty dirs. (Gnu's --recursive-unlink removes even those, may replace them with files or symlinks.)

Gnu options (also Mac and FreeBSD unless noted) also include:

• --one-file-system: when creating archive, stay on the volume(s) where the specified roots are located, and don't cross mountpoints

• --null: -T reads null-terminated names, ignores "-C/etc" entries

• -l, --check-links: (BSD for -c only) when creating or extracting, print a message if not all hard links are processed

• -S, --sparse: (not on BSD) when creating archive, handle sparse files efficiently

Some Gnu tar formats:

• in future, will default to -H posix/pax, which is the POSIX.1-2001 format
• v.1.13 - current (v1.26 released Mar 2011, still current in Mar 2012): defaults to -H gnu, which is not very different from the -H oldgnu format.
• -H ustar is the portable POSIX.1-1988 format, some limits including max 256 char filenames, max 8G filesize; supports special files, but not sparse files; also seems not to support xattrs and ACLs

Example of local copy:

sudo tar -cf- [--one-file-system] /source | sudo tar -xp[v]f- -C /target

The optional flags are:

• --one-file-system: Stay on the volume where /source is located. This may or may not be the behavior you desire. Note that this feature isn't available on BusyBox tar in any case, though it is available in some other tar implementations.
• -v: verbose

Example of using tar in cpio-style (for which, see below):

cd /source sudo find . [-xdev] -depth [-print0] | sudo tar -cf- -T- [--null] | sudo tar -xp[v]f- -C /target

The next examples add the -z flag, to compress output using gzip. If you have a fast connection, you may omit this; alternatively, you may use -j for bzip2. Example of copying to remote machine:

sudo tar -czf- [--one-file-system] /source | ssh root@machine "tar -xpz[v]f- -C /target"

Example of copying from remote machine:

ssh root@machine "tar -czf- [--one-file-system] /source" | sudo tar -xpz[v]f- -C /target

cpio

cpio operates in one of these four modes:

• -o -H newc: create archive on stdout (or -F archive), including filenames supplied from stdin (like tar -cf- -T-)

-H newc: (on Mac, -H sv4cpio and/or -c) specifies to use SVR4 ASCII format, this is required to create archives in BusyBox

• -i: receive archive from stdin (or -F archive), extract specified patterns underneath curdir (like tar -xf-)
• -t: (on some systems, may need -it) receive archive from stdin (or -F archive), list contents to stdout (like tar -tf-)
• -p /target: receive list of files from stdin (or -F archive), copy them underneath /target (like tar -cf- -T- | tar -xf- -C /target)

tar vs cpio:

• if you have includes/excludes, it's easier to use find than unfamiliar and non-portable options of Gnu tar
• cpio handles special files (block and char devices, fifos, etc); traditional tar (-H v7 format) didn't, also didn't store symbolic owner/group
• when extracting hard links, tar must always include the first
• tar and the -H newc cpio format handle 32-bit inodes; though some older cpio formats didn't
• the -H newc cpio format has max 1024 char filenames (though Gnu cpio can handle arbitrary lengths) and max 4G filesize; the portable -H ustar tar format has max 256 char filenames and max 8G filesize
• in some implementations, cpio wouldn't copy files over 2G properly; don't have details about which implementations/versions are so limited

Example of local copy:

cd /source sudo find . [-xdev] -depth [\! -path ./lost+found] [-print0] | sudo cpio -pdm[vul] [-0] /target

The flags are:

• -xdev: stay on the volume where /source is located; this may or may not be the behavior you desire
• \! -path ./lost+found: omit empty ./lost+found; ! only needs to be escaped for some shells, and this sequence may be repeated
• -print0 ... -0: permits processing filenames with embedded \ns, not available on BusyBox cpio or Mac
• -d: create leading directories, like mkdir -p
• -m: preserve file's original mtime
• -v: verbose
• -u: overwrite existing files, even if they're newer than files being copied
• -l: create hard links instead of copying. This may or may not be the behavior you desire. Note that this feature isn't available on BusyBox cpio in any case, though it is available in some other cpio implementations.

Other flags are:

• -oA -F archive: append to existing archive (not on BusyBox or BSD)
• -L: follow symlinks (not on BusyBox)
• --sparse: write files with blocks of zeros as sparse files (Gnu only)

Example of copying to remote machine:

cd /source sudo find . [-xdev] -depth [-print0] | sudo cpio -o -H newc [-0] | [gzip -3 |] ssh root@machine "cd /target; gunzip | cpio -idm[vu]"

rsync

rsync must be installed on both source and target machines. Example:

rsync -a[vzx] --delete [--numeric-ids] [-HAX] [--exclude=/proc --exclude=/dev --exclude=/sys] /source/ [root@machine:]/target

Note that the trailing / on /source/ is essential, if you want /target to end up a clone of /source.

The flags are:

• -v: verbose
• -z: compress data during transfer
• -x or --one-file-system: skip subdirectories on different volumes
• --numeric-ids: rsync's default is to use symbolic user/groupnames
• -H or --hard-links: preserve hard links in copied data, this can be expensive, and won't break existing hard links on /target unless rsync needed to write updated data to some of the linked files
• -A or --acls: preserve ACLs
• -X or --xattrs: preserve xattrs

• -8 or --8-bit-output: don't escape chars in filename even if they're invalid in current locale
• --partial: keep partially-transferred files

• --inplace: overwrite existing files in place
  rsync's default behavior is to write new files and move them into place when complete; this breaks any hard links the existing file may have had and makes a copy-on-write filesystem see the file as entirely new. --inplace specifies an alternate behavior, of writing updated data directly into the existing file.

• -h or --human-readable: report statistics in prettier form (use once for powers of 10, twice for powers of 2)
• --progress: show progress of each file transferred, implies -v
• -n or --dry-run

rsync's full manpage

rsync's advantages:

• like cp, can handle ACLs and xattrs; many tar and cpio implementations don't
• like cpio, can optionally work over the network
• can resume incomplete transfers, and can do incremental updates if the data is being copied multiple times

Warning: I've several times had it stop in the middle of a large copy because of some funky filenames, or corrupt packets. Using cpio or tar over ssh seemed to work fine. It may be more reliable to use one of the other methods for initial bulk copies, and restrict the use of rsync to incremental updates. Or it may be that the other methods are just silently ignoring issues that rsync more decently chooses to announce.

Including/excluding from transfers:

• --include=pattern[/]
• --exclude=pattern[/]

Each of these may be repeated. The patterns may contain *, **, ?, [a-z]. A trailing dir/*** matches both dir and its contents. If pattern contains a ** or (non-trailing) /, it matches against the full path from /target, else it only matches against the final path element.

lvm

"This is one reason I like LVM. Just add the new disk to the volume group, pvmove the logical volumes from the old to new disk, [wait a bit], remove the old disk from the volume group, and then from the system. If it's your boot disk you're replacing then you also need to update your boot loader."

See Setting up Logical Volumes with LVM.

Other tools

cd /target dump 0af - /source | restore rf -

cd /source pax -pe -rw -v [-X] -YZ . /target