Migrating data: Difference between revisions
Dubiousjim (talk | contribs) (Break up into sections) |
Dubiousjim (talk | contribs) (remove {{Draft}}, mark Warning as Todo) |
||
Line 1: | Line 1: | ||
We compare several methods for copying/migrating large amounts of data. The aim is to preserve all file permissions and metadata, which will require root access on both source and target filesystems. | We compare several methods for copying/migrating large amounts of data. The aim is to preserve all file permissions and metadata, which will require root access on both source and target filesystems. | ||
Line 232: | Line 231: | ||
{{Warning | {{Todo| Warning: I had it stop in the middle of a large copy because of some funky filenames; cpio handled them fine. I had assumed these filenames included <code>\n</code>s, but allegedly rsync should be fine with those. Now I've lost track of which files it was.}} | ||
Revision as of 14:58, 28 March 2012
We compare several methods for copying/migrating large amounts of data. The aim is to preserve all file permissions and metadata, which will require root access on both source and target filesystems.
We include information about some options that aren't available on BusyBox's implementation of these tools, because you may be copying data to/from systems with other Unix tools installed, which do provide those options.
Bit-copying methods
If your source data occupies an entire partition (that is, nothing else resides on that partition), and your target partition is the same size or larger, and you're willing to have it use the same filesystem as the source, then one option is to bit-copy the whole source partition:
Boot from LiveCD and make sure that neither /dev/sourcepart nor /dev/targetpart are mounted.
Example:
dd bs=10M if=/dev/sourcepart of=/dev/targetpart
This also copies unoccupied blocks on the sourcepart to the targetpart. A more optimal variant is to use partimage, which copies only the used portions of the sourcepart. One limitation is that currently (v0.6.9, released July 2010, current as of Mar 2012) partimage doesn't support ext4 format.
Once you've finished, the filesystem on targetpart may occupy less than the total space available on targetpart. You may want to use a tool like resize2fs to grow the partition.
Higher-level methods
The other options we'll discuss suppose that the target filesystem is already formatted and mounted as /target.
If you're proposing to copy volatile data from a running system, you may want to copy from a lvm snapshot, or do the copy from single-user mode, or from a LiveCD or other boot medium so that your source data is no longer in use during the copy. If you're just proposing to copy from /home or such, no such precautions are necessary.
If you're copying a complete filesystem, you'll probably only want to copy the /sys and /proc mountpoints, but not their current contents. Remember to copy /boot and /dev and /tmp, the last two of which may require special attention. Pay attention to other mountpoints (such as /mnt, /media) and RAM-based filesystems (such as /run and /lib/rc/init.d).
mkdir /tmp/dev mount --move /dev /tmp/dev copy /dev to /mnt/sdb5 using one of the methods elsewhere on this page mount --move /tmp/dev /dev rmdir /tmp/dev
cp
Depending on what system you're working with, and what kind of data you're copying, you may be able to do a satisfactory local copy using just cp
. But there are a number of limits here that prevent this from being a general solution.
The Gnu implementation of cp
provides the following options (among others):
-a
: same as-RP -p/--preserve=mode,ownership,timestamps,links,context,xattr
-v
: verbose-l
: make hard links instead of copying-s
: make symlinks instead of copying (source file names must be absolute)--sparse=always
-u
,--update
: don't copy files or symlinks over an existing file unless its mtime is older-x
,--one-file-system
: skip subdirectories on different volumes
The BusyBox version provides all of the Gnu options except --sparse
and -u
and -x
, and may lack the context
and xattr
options to --preserve
. It will preserve hard links: if /source/A and /source/B are hard links, /target/A and /target/B will also be hard links with each other (though not with the originals). BusyBox's cp
seems to silently ignore the -v
option.
The FreeBSD implementation provides all of these except -s
and --sparse
and -u
, and the --preserve
options from links
on: in particular, it always copies hard links as separate files. The Mac version shares the limitations of the FreeBSD version, and additionally lacks -x
. Also, on the Mac version the -a
shortcut isn't available, you must explicitly say -RPp
.
-r
option. Its behavior varies between implementations, and may diverge from what you want for special files or symlinks.
tar
Options recognized by BusyBox and other implementations:
-c
: create a new archive, records existing perms and mtimes, symbolic owner/group (can specify--numeric-owner
), hard links among archive elements-x
: extract files from an archive, uses current owner/group unless root, subtracts current umask unless root or-p
, when extracting hard links must include first-t
: list the contents of an archive
-f archive
: use-f -
for stdin/stdout
- Note: On some but not all implementations, the archive defaults to stdin/stdout when
-f
is not supplied. It's most portable to always declare this explicitly.
-O
: extract files to standard output-C dir
: change to directory dir, when creating this is order-sensitive-v
: verbose
-z
: compress archive with gzip-j
: compress archive with bzip2-Z
: compress archive with compress
-h
: follow symlinks, archive and extract the files they point to (like-L
in other programs)-m
: don't extract file's original mtime, leave touched with the extraction time-T file
: get names to extract or create from file, which can also include "-C/etc" lines--exclude=pattern
: exclude specified glob patterns-X file
: exclude glob patterns listed in file
- Note: Default globbing rules: (i) matches after any /, not just at ^; (ii) case-sensitive; (iii) wildcards match /
The following common options are allegedly also honored by BusyBox (according to
comments in its source) but aren't declared in its --help
:
-o
,--no-same-owner
: (BSD doesn't recognize a long option) extract as yourself (default for ordinary users)
-p
,--same-permissions
: (BSD doesn't recognize this long option) extract permissions verbatim, instead of subtracting current umask (usually default for root)
--no-same-permissions
: subtract umask from extracted permissions (default for ordinary users)
--numeric-owner
: create using numbers for user/groups
-k
: don't replace existing files when extracting
--overwrite
: (not on BSD) more aggressively overwrite existing files and directory metadata when extracting
tar's default behavior is to first remove existing files (otherwise, all their hardlinks would be modified) and symlinks. If an existing dir is nonempty, its existing contents won't be removed; tar will instead just update the dir's metadata.--overwrite
instead follows existing symlinks, and removes anything that impedes extraction except non-empty dirs. (Gnu's--recursive-unlink
removes even those, may replace them with files or symlinks.)
Gnu options (also Mac and FreeBSD unless noted) also include:
--one-file-system
: when creating archive, stay on the volume(s) where the specified roots are located, and don't cross mountpoints
--null
:-T
reads null-terminated names, ignores "-C/etc" entries
-l
,--check-links
: (BSD for-c
only) when creating or extracting, print a message if not all hard links are processed
-S
,--sparse
: (not on BSD) when creating archive, handle sparse files efficiently
Some Gnu tar formats:
- in future, will default to
-H posix/pax
, which is the POSIX.1-2001 format - v.1.13 - current (v1.26 released Mar 2011, still current in Mar 2012): defaults to
-H gnu
, which is not very different from the-H oldgnu
format. -H ustar
is the portable POSIX.1-1988 format, some limits including max 256 char filenames, max 8G filesize; supports special files, but not sparse files; also seems not to support xattrs and ACLs
Example of local copy:
sudo tar -cf- [--one-file-system] /source | sudo tar -xp[v]f- -C /target
The optional flags are:
--one-file-system
: Stay on the volume where /source is located. This may or may not be the behavior you desire. Note that this feature isn't available on BusyBoxtar
in any case, though it is available in some othertar
implementations.-v
: verbose
Example of using tar in cpio-style (for which, see below):
cd /source sudo find . [-xdev] -depth [-print0] | sudo tar -cf- -T- [--null] | sudo tar -xp[v]f- -C /target
The next examples add the -z
flag, to compress output using gzip. If you have a fast connection, you may omit this; alternatively, you may use -j
for bzip2. Example of copying to remote machine:
sudo tar -czf- [--one-file-system] /source | ssh root@machine "tar -xpz[v]f- -C /target"
Example of copying from remote machine:
ssh root@machine "tar -czf- [--one-file-system] /source" | sudo tar -xpz[v]f- -C /target
cpio
cpio operates in one of these four modes:
-o -H newc
: create archive on stdout (or-F archive
), including filenames supplied from stdin (liketar -cf- -T-
)
-H newc
: (on Mac,-H sv4cpio
and/or-c
) specifies to use SVR4 ASCII format, this is required to create archives in BusyBox
-i
: receive archive from stdin (or-F archive
), extract specified patterns underneath curdir (liketar -xf-
)-t
: (on some systems, may need-it
) receive archive from stdin (or-F archive
), list contents to stdout (liketar -tf-
)-p /target
: receive list of files from stdin (or-F archive
), copy them underneath /target (liketar -cf- -T- | tar -xf- -C /target
)
tar vs cpio:
- if you have includes/excludes, it's easier to use
find
than unfamiliar and non-portable options of Gnu tar - cpio handles special files (block and char devices, fifos, etc); traditional tar (
-H v7
format) didn't, also didn't store symbolic owner/group - when extracting hard links, tar must always include the first
- tar and the
-H newc
cpio format handle 32-bit inodes; though some older cpio formats didn't - the
-H newc
cpio format has max 1024 char filenames (though Gnu cpio can handle arbitrary lengths) and max 4G filesize; the portable-H ustar
tar format has max 256 char filenames and max 8G filesize - in some implementations, cpio wouldn't copy files over 2G properly; don't have details about which implementations/versions are so limited
Example of local copy:
cd /source sudo find . [-xdev] -depth [\! -path ./lost+found] [-print0] | sudo cpio -pdm[vul] [-0] /target
The flags are:
-xdev
: stay on the volume where /source is located; this may or may not be the behavior you desire\! -path ./lost+found
: omit empty ./lost+found;!
only needs to be escaped for some shells, and this sequence may be repeated-print0
...-0
: permits processing filenames with embedded\n
s, not available on BusyBoxcpio
or Mac-d
: create leading directories, likemkdir -p
-m
: preserve file's original mtime-v
: verbose-u
: overwrite existing files, even if they're newer than files being copied-l
: create hard links instead of copying. This may or may not be the behavior you desire. Note that this feature isn't available on BusyBoxcpio
in any case, though it is available in some othercpio
implementations.
Other flags are:
-oA -F archive
: append to existing archive (not on BusyBox or BSD)-L
: follow symlinks (not on BusyBox)--sparse
: write files with blocks of zeros as sparse files (Gnu only)
Example of copying to remote machine:
cd /source sudo find . [-xdev] -depth [-print0] | sudo cpio -o -H newc [-0] | [gzip -3 |] ssh root@machine "cd /target; gunzip | cpio -idm[vu]"
rsync
rsync
must be installed on both source and target machines. Example:
rsync -a[vzx] --delete [--numeric-ids] [-HAX] [--exclude=/proc --exclude=/dev --exclude=/sys] /source/ [root@machine:]/target
Note that the trailing / on /source/ is essential, if you want /target to end up a clone of /source.
The flags are:
-v
: verbose-z
: compress data during transfer-x
or--one-file-system
: skip subdirectories on different volumes--numeric-ids
: rsync's default is to use symbolic user/groupnames-H
or--hard-links
: preserve hard links in copied data, this can be expensive, and won't break existing hard links on /target unless rsync needed to write updated data to some of the linked files-A
or--acls
: preserve ACLs-X
or--xattrs
: preserve xattrs
-8
or--8-bit-output
: don't escape chars in filename even if they're invalid in current locale--partial
: keep partially-transferred files
--inplace
: overwrite existing files in place
rsync's default behavior is to write new files and move them into place when complete; this breaks any hard links the existing file may have had and makes a copy-on-write filesystem see the file as entirely new.--inplace
specifies an alternate behavior, of writing updated data directly into the existing file.
-h
or--human-readable
: report statistics in prettier form (use once for powers of 10, twice for powers of 2)--progress
: show progress of each file transferred, implies-v
-n
or--dry-run
rsync's advantages:
- like cp, can handle ACLs and xattrs; many tar and cpio implementations don't
- like cpio, can optionally work over the network
- can resume incomplete transfers, and can do incremental updates if the data is being copied multiple times
\n
s, but allegedly rsync should be fine with those. Now I've lost track of which files it was.
lvm
"This is one reason I like LVM. Just add the new disk to the volume group, pvmove the logical volumes from the old to new disk, [wait a bit], remove the old disk from the volume group, and then from the system. If it's your boot disk you're replacing then you also need to update your boot loader."
See Setting up Logical Volumes with LVM.
Other tools
cd /target dump 0af - /source | restore rf -
cd /source pax -pe -rw -v [-X] -YZ . /target