Svn2Svn – Copy and Sync Between SVN Repositories

The Mission

Break a huge and poorly organized Subversion repository into a number of small, project oriented repositories.

The Challenge

1. The repository is so large that generating a dump is taking forever.
2. There are build systems connecting to the current repository.
3. During the migration process, we should not stop developer from committing their code.

Traditional Tools

The typical solution is to use traditional subversion dump and filter tools. But that doesn’t work for us due the the size of the repository. Dump takes long time and filter into many many small repositories take weeks. In the mean time, developer need to continue to commit and build system must continue to work. Hence we need a solution that enable us to migrate gradually.

Replay Tool

The solution to the problem is a subversion replay tool. The idea is to pull the revision history of a subtree in source repository, replay it back to the destination repository. New commits to the source repository can be incrementally copied over. It should work like svnsync but for a subtree instead of entire repository, and you don’t have to start from revision zero. The benefits of such tool are:

1. Projects can be moved one by one or a few at a time.
2. Each project can move according to its own agenda.
3. The build system can switch to new repository first while developers still committing to old repository.

Why Another Svn2Svn

There was already such a tool available at http://svn2svn.codeplex.com. That tool has its own advantages but doesn’t work for us because it

1. Does not preserve the author and date/time of the revision.
2. Does not copy node properties, e.g. svn:ignore, svn:external.
3. Does not handle copied or moved nodes. This is particularly an issue for tags.
4. Fails at a few other edge cases

So I set out to completely redesign and rewrite another Svn2Svn.

The New Svn2Svn

The new Svn2Svn is now born. It runs on Windows platform and is based on SharpSvn. It fulfills above requirement and has below features.

• Support both command line and Windows UI.
• Copies change sets from one SVN repository to another.
• Supports non-rooted path (subtree) for both source and/or destination.
• Doesn't require source and destination to have same path.
• Copies node properties so that properties like svn:ignore and svn:external are preserved in the destination repository.
• Copies revision properties so that author and date/time of revisions are preserved in destination repository. [1]
• Doesn't require to starting from zero revision.
• Can optionally specify a source revision range.
• Supports move and copy in addition to add/delete/modify. A tag in source is copied as a true tag in destination with accurate copy from path and revision.
• Able to gracefully stop the process in the middle.
• It automatically resumes from a previously stopped session, either manually stopped or due to an error.
• Able to auto-resync and incrementally copy over new change sets from source. [1]
• Properly handle rename that only changes the case of letter. E.g, rename “Abc” to “abc”.
• Intuitive to use. When fails, you get detailed error message.
• In most of time, you are able to resume from failure by just deleting your working directory and restart with the same parameters.

Note [1] features require destination repository support revision property editing.

Download and Installation

Svn2Svn is a 32 bit .Net application packed in a zip file that contains both Windows UI and command line executable. The distribution is bundled with SharpSvn DLLs, so you can easily unzip to any folder you like and run one of two executable files. They run on 64bit system as 32bit applications.

To obtain Svn2Svn, downloaded latest version from http://code.google.com/p/kennethxublogsource/downloads/list.

Windows UI

Command Line

C:\Svn2Svn>Svn2SvnConsole.exe

Usage:

        Svn2SvnConsole.exe [options] sourceUri destinationUri workingDir

options:

        -R:from:to Copy revisions specified by from and to.

        -R:start   Copy revisions from specified start to HEAD.

        -R::end    Copy revisions from 0 to specified end.

        -X         Do not copy any revision property.

        -X:[ADR]   Do not copy one or more revision properties. e.g.:

                   -X:D  - Do not copy date/time revision property.

                   -X:AD - Do not copy author and date/time revision property.

        -I         Ignore all none fatal errors. Only log them.

        -V         Log every revision.

        -V+        Log every revision and node.

Using Svn2Svn

Svn2Svn UI and command line provides same set of features, we’ll use command line as example to show the usage.

Destination Repository Configuration

Some important features of Svn2Svn requires the destination repository to allow revision property editing. Very often you’ll get below error message:

Error Processing Revision

Repository has not been enabled to accept revision propchanges;

ask the administrator to create a pre-revprop-change hook

This is because the destination repository wasn’t configured to allow revision property editing. To enable revision property editing, you need to provide pre-revprop-change hook. And make sure the hook will allow the user who is running this tool to edit any property. The Svn2Svn zip package also enclosed a file named “pre-revprop-change.exe” in the “Misc” folder. It is very useful if you use a local repository as your destination. The pre-revprop-change.exe allows free editing of revision properties by anybody and it prevents a DOS window from being popped up on every commit, which happens when you use a .bat or .cmd based hook.

Typical Usage

The tool is quite intuitive to use. In addition to source and destination URI which are very obvious, it needs a working directory to manipulate the local changes. Below is a typical example:

Svn2SvnConsole.exe http://sourcerepo/svn/project1 http://destinationrepo/svn/ c:\temp\project1

This command can be scheduled to run many times. Every time it runs, it will bring new commits from source to destination. Be careful that you should not make conflicting changes in the destination repository if you intended to run the same command again to bring new commits from source.

Specify Revision Range

If you’ll need to specify (or have to specify) the range of revisions to copy over to destination, use the –r option. You can specify both starting and ending revisions or just one of them. If only one is specified, the other will take a default value. The default value is zero for the starting revision and HEAD for the ending revision. Command below copies from revision zero to revision 678.

Svn2SvnConsole.exe http://sourcerepo/svn/project1 http://destinationrepo/svn/ c:\temp\project1 –r::678

Control Copy of Revision Property

There are times that you have no control over the destination repository so you cannot change the revision property. In this case, you’ll need to disable copying of the revision property by using –X option.

You can also selectively disable copying of specific revision property by using option –X:  immediately followed by any combination of letter A, D and R. For example, option  –X:AD disables copying of author and date/time.

• A – Author, svn:author
• D – Date/time, svn:date
• R – Source revision, svn2svn:revision

Logging Control

Use –V option to have the tool output message for every revision copied. Use -V+ option to further output the detail process information of each node.

Error Handling

Whenever a non-fatal error occurs, the tool will display a title followed by the error message and give you options to fail, retry, ignore or ignore all.

• Fail will stop the process, print out the full stack trace. Command line tool will exit with non-zero error code.
• Retry will attempt the same operation again.
• Ignore will leave the problem as is and move on to next operation.
• Ignore all will do the same as “ignore” and additionally have the tool automatically ignore all further errors with same title.

The command line tool also support option –I to automatically ignore all non-fatal errors. Non-fatal errors are those errors related only to a specific revision or node.

Interrupting the Process

Windows UI tool turns the “Copy” button into “Stop” button when the copy is in progress. You can click on Stop button to stop the process. Command line utility supports Ctrl-C to stop the process, as soon as you press Ctrl-C during the copy process, the tool will ask you if you want to gracefully stop the process or fail immediately. You should almost always gracefully stop the process by pressing ‘Y’ key. The fail immediate option is provided only for the cases when the graceful stop doesn’t work. If that happens, press ‘X’ key to exit immediately. If you pressed Ctrl-C by mistake, you can press any key other than ‘X’ and ‘Y’ to resume copy process.

How Svn2Svn Works

Every time you run Svn2Svn, it goes through below steps:

1. Determine the highest revision of the source if end revision wasn’t specified.
2. Create the destination directory if one doesn’t already exist in repository.
3. If working directory doesn’t exist, check out the destination to the working directory. Otherwise, do svn cleanup, then revert back pending changes if any and delete all non versioned files and directories, finally do svn update.
4. Scan through revision history of destination to detect the previous copy information by retrieving the revision property named “svn2svn:revision”. And establish the source revision to destination revision mapping for anything that was previously copied.
5. Loop through source revisions a hundred at a time to avoid http timeout. Skip through those already copied revisions using revision map collected in the previous step.
6. If this is the first time copying, i.e. there is no revision mapping discovered in step 4, the tool exports the nodes of first source revision, including their properties, to working directory and commit the changes to destination.
7. For each subsequent revisions, looping through all node in the change set and perform below operation:
   1. Delete, modify or add/copy the node.
   2. Delete existing properties from node and copy the node properties from source.
8. If a letter case change only renamed is detected, the add/copy operations are all deferred. The tool executes an additional commit followed by deferred add/copy operations.
9. When every nodes are done, commit to the destination repository with the same message from source.
10. Edit the revision repository according the the options specified. You typically want the author and date/time copied over.
11. Add “svn2svn:revision” property to the destination revision, with value being the source revision number. This information is essential to incremental copy the change set.
12. Go back to step 6 and repeat the process until the end revision is reached.

Misc. Tools

The Svn2Svn package also include addition stuff in Misc sub directory. One is the “pre-revprop-change.exe” file to enable the revision property editing that we have discussed. Another is a zip of empty Subversion 1.6 repository for you to easily create an clean 1.6 repository. Simply unzip it to any directory you choose.

Known Issues and Workarounds

Authentication

At this moment, the tool doesn’t have build in authentication mechanism. The work around is to use any other SVN client to authentication with the source and/or target and make sure you save the password. I have used TortoiseSVN and standard SVN command line client with success. If somebody have a patch to enable authentication, I’m more then happy to put it in but I don’t have a need for this.

Error in Working Directory

From time to time, you may encounter errors. Some of them are due to a bug in this tool, a bug in SVN itself or nature of case-sensitively difference between Windows vs. SVN repo. But so far most of problems can be solved by simply deleting the working directory (delete the directory itself, not just the content in it), and rerunning the same command again.