Transferring data: Difference between revisions

[https://en.wikipedia.org/wiki/Rsync Rsync] is a popular tool for ensuring that two separate datasets are the same but can be quite slow if there are a lot of files or there is a lot of latency between the two sites, i.e. they are geographically apart or on different networks. Running rsync will check the modification time and size of each file, and will only transfer the file if one or the other does not match. If you expect modification times not to match on the two systems you can use the "-c" option, which will compute checksums at the source and destination, and transfer only if the checksums do not match.  

Some users have encountered trouble when using <code>rsync</code> to transfer directories into <code>/project</code>. The <code>-p</code> and <code>-g</code> flags (or equivalently <code>--perms</code> and <code>--group</code>) may cause the group ownership of files to be incorrectly set. Since quotas in <code>/project</code> are enforced based on group ownership this in turn may lead to <code>Disk quota exceeded</code>. The frequently-used <code>-a</code> or <code>--archive</code> flag implies <code>-p</code> and <code>-g</code>. Best advice is to use Globus instead of rsync, but if you choose to use rsync, do not use <code>-p</code> and <code>-g</code>.

Therefore '''<code>rsync -rltv ...</code>''' is a good replacement for <code>rsync -av ...</code> when syncing files into the <code>/project</code> filesystem.