Start to wrap up all thoughts on what should happen for rsync 3.

rsync3.txt

+-*- indented-text -*-
+
+Notes towards a new version of rsync 
+Martin Pool <mbp@samba.org>
+
+
+Good things about the current implementation:
+
+  - Widely known and adopted.
+
+  - Fast/efficient, especially for moderately small sets of files over
+    slow links (transoceanic or modem.)
+
+  - Fairly reliable.
+
+  - The choice of runnning over a plain TCP socket or tunneling over
+    ssh.
+
+  - rsync operations are idempotent: you can always run the same
+    command twice to make sure it worked properly without any fear.
+    (Are there any exceptions?)
+
+  - Small changes to files cause small deltas.
+
+  - There is a way to evolve the protocol to some extent.
+
+  - rdiff and rsync --write-batch allow generation of standalone patch
+    sets.  rsync+ is pretty cheesy, though.  xdelta seems cleaner.
+
+  - Process triangle is creative, but seems to provoke OS bugs.
+
+  - "Morning-after property": you don't need to know anything on the
+    local machine about the state of the remote machine, or about
+    transfers that have been done in the past.
+
+  - You can easily push or pull simply by switching the order of
+    files.
+
+
+Bad things about the current implementation:
+
+  - Persistent and hard-to-diagnose hang bugs remain
+
+  - Protocol is sketchily documented, tied to this implementation, and
+    hard to modify/extend
+
+  - Both the program and the protocol assume a single non-interactive
+    one-way transfer 
+
+  - A list of all files are held in memory for the entire transfer,
+    which cripples scalability to large file trees
+
+  - Opening a new socket for every operation causes problems,
+    especially when running over SSH with password authentication.
+
+  - Renamed files are not handled: the old file is removed, and the
+    new file created from scratch.
+
+  - The versioning approach assumes that future versions of the
+    program know about all previous versions, and will do the right
+    thing.
+
+  - People always get confused about ':' vs '::'
+
+  - Error messages can be cryptic.
+
+
+Protocol philosophy:
+
+   *The* big difference between protocols like HTTP, FTP, and NFS is
+    that their fundamental operations are "read this file", "delete
+    this file", and "make this directory", whereas rsync is "make this
+    directory like this one".
+
+
+Questionable features:
+
+  These are neat, but not necessarily clean or worth preserving.  
+
+  - The remote rsync can be wrapped by some other program, such as in
+    tridge's rsync-mail scripts.  The general feature of sending and
+    retrieving mail over rsync is good, but this is perhaps not the
+    right way to implement it.
+
+
+Desirable features:
+
+  These don't really require architectural changes; they're just
+  something to keep in mind.
+ 
+  - Synchronize ACLs and extended attributes
+
+  - Anonymous servers should be efficient
+
+  - Code should be portable to non-UNIX systems
+
+  - Should be possible to document the protocol in RFC form
+
+  - --dry-run option
+
+  - IPv6 support.  Pretty straightforward.
+
+  - Allow the basis and destination files to be different.  For
+    example, you could use this when you have a CD-ROM and want to
+    download an updated image onto a hard drive.
+
+  - Efficiently interrupt and restart a transfer.  We can write a
+    checkpoint file that says where we're up to in the filesystem.
+    Alternatively, as long as transfers are idempotent, we can just
+    restart the whole thing.  [NFSv4]
+
+  - Scripting support.  
+
+  - Propagate atimes and do not modify them.  This is very ugly on
+    Unix.  It might be better to try to add O_NOATIME to kernels, and
+    call that.
+
+  - VFS.  Useful?
+
+  - Unicode.  Probably just use UTF-8 for everything.
+
+
+Hard links:
+
+  At the moment, we can recreate hard links, but it's a bit
+  inefficient: it depends on holding a list of all files in the tree.
+  Every time we see a file with a linkcount >1, we need to search for
+  another known name that has the same (fsid,inum) tuple.  We could do
+  that more efficiently by keeping a list of only files with
+  linkcount>1, and removing files from that list as all their names
+  become known.
+
+
+Scripting issues:
+
+  - Perhaps support multiple scripting languages: candidates include
+    Perl, Python, Tcl, Scheme (guile?), sh, ...
+
+  - Simply running a subprocess and looking at its stdout/exit code
+    might be sufficient, though it could also be pretty slow if it's
+    called often.
+
+  - There are security issues about running remote code, at least if
+    it's not running in the users own account.  So we can either
+    disallow it, or use some kind of sandbox system.
+
+
+Scripting hooks:
+
+  - Whether to transfer a file
+
+  - What basis file to use
+
+  - Logging
+  
+  - Whether to allow transfers (for public servers)
+
+  - Authentication
+
+  - Locking
+
+
+Interactive interface:
+
+  - Something like ncFTP, or integration into GNOME-vfs.  Probably
+    hold a single socket connection open.
+
+  - Can either call us as a separate process, or as a library.
+
+  - The standalone process needs to produce output in a form easily
+    digestible by a calling program, like the --emacs feature some
+    have. 
+
+  - Yow!  emacs support.  (You could probably build that already, of
+    course.)
+
+
+Pie-in-the-sky features:
+
+  These might have a severe impact on the protocol, and are not
+  clearly in our core requirements.  It looks like in many of them
+  having scripting hooks will allow us 
+
+  - Transport over UDP multicast.  The hard part is handling multiple
+    destinations which have different basis files.  We can look at
+    multicast-TFTP for inspiration.
+
+  - Conflict resolution.  Possibly general scripting support will be
+    sufficient.
+
+  - Integrate with locking.  It's hard to see a good general solution,
+    because Unix systems have several locking mechanisms, and grabbing
+    the lock from programs that don't expect it could cause deadlocks,
+    timeouts, or other problems.  Scripting support might help.
+
+  - Replicate in place, rather than to a temporary file.  This is
+    dangerous in the case of interruption, and it also means that the
+    delta can't refer to blocks that have already been overwritten.
+    On the other hand we could semi-trivially do this at first by
+    simply generating a delta with no copy instructions.
+
+  - Replicate block devices.  Most of the difficulties here are to do
+    with replication in place, though on some systems we will also
+    have to do I/O on block boundaries.
+
+
+In favour of evolving the protocol:
+
+  - Keeping compatibility with existing rsync servers will help with
+    adoption and testing.
+
+  - We should at the very least be able to fall back to the new
+    protocol.
+
+  - Error handling is not so good.
+
+
+In favour of using a new protocol:
+
+  - Maintaining compatibility might soak up development time that
+    would better go into improving a new protocol.
+
+  - If we start from scratch, it can be documented as we go, and we
+    can avoid design decisions that make the protocol complex or
+    implementation-bound. 
+
+
+Error handling:
+
+  - Errors should come back reliably, and be clearly associated with
+    the particular file that caused the problem.
+
+  - Some errors ought to cause the whole transfer to abort; some are
+    just warnings.  If any errors have occurred, then rsync ought to
+    return an error.
+
+
+Concurrency:
+
+  - We want to keep the CPU, filesystem, and network as full as
+    possible as much of the time as possible.
+
+  - We can do nonblocking network IO, but not so for disk.
+
+  - It makes sense to on the destination be generating signatures and
+    applying patches at the same time. 
+
+  - Can structure this with nonblocking, threads, separate processes,
+    etc.
+
+
+Uses:
+
+  - Mirroring software distributions:
+
+  - Synchronizing laptop and desktop
+
+  - NFS filesystem migration/replication.  See
+    http://www.ietf.org/proceedings/00jul/00july-133.htm#P24510_1276764
+
+  - Sync with PDA
+ 
+  - Network backup systems
+
+  - CVS filemover
+
+
+Conflict resolution:
+
+  - Requires application-specific knowledge.  We want to provide
+    policy, rather than mechanism.
+
+  - Possibly allowing two-way migration across a single connection
+    would be useful.
+
+
+Moved files:
+
+  - There's no trivial way to detect renamed files, especially if they
+    move between directories.
+
+  - If we had a picture of the remote directory from last time on
+    either machine, then the inode numbers might give us a hint about
+    files which may have been renamed.
+
+  - Files that are renamed and not modified can be detected by
+    examining the directory listing, looking for files with the same
+    size/date as the origin.
+
+
+Filesystem migration:
+
+  The NFSv4 working group wants atomic migration.  Most of the
+  responsibility for this lies on the NFS server or OS.  
+
+  If migrating a whole tree, then we could do a nearly-atomic rename
+  at the end.  This ties in to having separate basis and destination
+  files.
+
+  NFSv4 probably wants to migrate file locks, but that's not really
+  our problem.
+
+
+Scalability:
+ 
+  We should aim to work well on machines in use in a year or two.
+  That probably means transfers of many millions of files in one
+  batch, and gigabytes or terabytes of data.
+
+  For argument's sake: at the low end, we want to sync ten files for a
+  total of 10kb across a 1kB/s link.  At the high end, we want to sync
+  1e9 files for 1TB of data across a 1GB/s link.
+
+  On the whole CPU usage is not normally a limiting factor, if only
+  because running over SSH burns a lot of cycles on encryption.
+
+
+Streaming:
+
+  A big attraction of rsync is that there are few round-trip delays:
+  basically only one to get started, and then everything is
+  pipelined.  This is a problem with FTP, and NFS (at least up to
+  v3).  NFSv4 can pipeline operations, but building on that is
+  probably a bit complicated.