Category Archives: JavaRanch

migrating coderanch from svn to git

Posted on by
1

CodeRanch has been using SVN for a long time for the forum software. It’s high time to switch. We have just over 20 projects in our one SVN repository. Most are small/inactive so it wouldn’t be terrible to have the history in only the biggest project. However, I wanted to try to do it “right” and migrate the history of each project into a separate git repository.

Also see:

Choosing GitLab as a provider

We chose to use GitLab instead of GitHub for a few reasons

  1. GitLab has free organizations (security grouping) for private repositories. It also allows multiple admins. (which you can’t do in a free github repo)
  2. GitLab has a built in continuous integration tool (we aren’t using that yet, but want to leave the option open.)
  3. A couple moderators use GitLab professionally and have had good experiences with it.


Getting a local dump of the SVN database

Migrating a remote repository involves a large number of network roundtrips. It’s far faster to export the SVN repository/database to a local dump file. Starting with SVN 1.7, this is a completely client side operation.

However, I’ve been using SVN through Eclipse so I didn’t actually have a SVN 1.7 command line installed. The first thing I did was install a command line of SVN 1.7:

brew install subversion

Then I got a dump of the whole repository. I used svnrdump so I didn’t need to sign on to the machine with the repo. I ran:

svnrdump dump '<url>' > full.dmp

We have just under ten thousands revisions so the dump took about 20 minutes. The full dump was just under 800MB. (A lot of this is duplication in tags. The GitLab repository is about a quarter the size.)

Create authors.txt

I decided to do it by hand. I looked at the conf/htaccess-projects.acl SVN file since I have admin access to the server. There were 90 users. I wound up migrating them all (which was a poor decision.)

This hit or miss approach works (slowly) because “git svn clone” complains if it encounters an author that isn’t defined. This let me go back and add that person without having to manually do a lot of analysis. Luckily, “git svn clone” does let you resume from where you left off.

Author: xxx not defined in authors.txt file

Use git-svn bridge

I used the git-svn bridge for the actual migration. This was only a few steps for the small projects that didn’t have branches or tags:

  1. Use the git-svn bridge to clone just the part of the repo we want: git svn clone <url>/svn/project/ -A authors.txt new-git-repo
  2. Add an origin on my machine: git remote add origin git@gitlab.com:coderanch/new-repo.git (fun fact, you don’t need to create the repository on gitlab. When you push, it gets created for you.)
  3. Push all: git push -u origin –all
  4. Push tags (we don’t have any for most projects so skipped this): git push –tags
  5. Repeat for all our projects. (We have about 20)

It turned out some of the projects in our repo have trunk/branches/tags inside. So I needed the command to use the standard layout (the -s flag):

git svn clone -s https://svn.javaranch.com/svn/project -A authors.txt new-repo

For branches

We hardly have any branches so I didn’t search for the optimal way. I used a three step procedure.

  1. I used the UNIX script from sailmaker to convert the SVN tags to Git tags. 
    for branch in `git branch -r | grep "branches/" | sed 's/ branches\///'`; 
do   
   git branch $branch refs/remotes/$branch 
done
  2. Checked out each branch 
    git checkout -b <branch_name>
  3. Sent it on to the remote repo: git push –all

This approach leaves stray remote branches. They are visible using git branch -a, but not in the GitLab UI. They seem to go away if you clone the repository to another directory so I’m thinking they are local branches and were never pushed to GitLab.

For the big repo with tags

We have 396 tags for the forum software and zero tags for all our other projects. I tried migrating our entire repository including tags using “git svn clone.” It took too long. 9K+ commits 800MB, 300+ tags is really slow. After running (against my local SVN) for over 24 hours, it was less than half done.

I realized the bulk of the time was going to migrating tags. So I did the “git svn clone” migration for just the trunk and branches. Then I dealt with the tags using a script.

First I created a local repo since this was the big one. This took about 20 minutes to import:

  • svnadmin create jforum-local-svn
  • svnadmin load jforum-local-svn < full.dmp

Then I did the clone:

git svn clone file:///<local repo> -A authors.txt 
  --trunk=JForum --branches=branches 
  jforum-from-local-dump-with-trunk-and-branches

Finally, I was ready to add gitlab as a remote. Again, I migrated the branches manually since there weren’t a lot.

Then I dealt with the tags. But that warrants a separate blog post.

Another possibility

There’s a number of tools called svn2git. The most promising one looks like this one per this post.

I didn’t try it because I was almost done at that point. Also, it requires you to build from source. Wasn’t worth the effort.

References

  • https://john.albin.net/git/convert-subversion-to-git
  • https://www.mugo.ca/Blog/Splitting-a-Subversion-repository-into-multiple-repositories
  • https://daneomatic.com/2010/11/01/svn-to-multiple-git-repos/
  • https://git-scm.com/docs/git-svn
  • https://www.getdonedone.com/converting-5-year-old-repository-subversion-git/ – uses master/trunk/branches structure – why want that?
  • https://github.com/nirvdrum/svn2git – uses git-svn bridge and does cleanup after. so presumably the same performance issue

migrating tags from a large coderanch repository from svn to git

Posted on by
Reply

To review, this repository has just under ten thousand commits and just under 400 tags. Migrating with “git svn clone” would have taken over 48 hours. Since the majority of the time was going to migrating the tags, I decided to migrate just the trunk/branches and then re-create the tags. After all they are just labels. This post describes the procedure.

Also see

Looking at the commit comments

Each migrated commit in git has a comment like:

git-svn-id: https://<url>/svn/projectName@8848 9a30da7b-550c-0410-b2c9-c7485123b453

This is good. It lets me map SVN commits (ex: 8848) to Git has (9a30da). This means it is possible to create the tags with a script.

Some “light googling” didn’t find such a script so I wrote my own.

Step 1 – Create a file containing all the SVN tag names to migrate

This is easy. Go to https://<svn url>/svn/tags/ in a browser and it gives you such a list. Just copy this to a file. My file had 396 lines in it. But copy/paste doesn’t charge per line so this was ok!

Step 2 – Map SVN tag names to SVN commit numbers and Git commit hashes

I wrote a script to create CSV of these pieces of informatio. I didn’t include creating the tags in that script so I could review the output in between. I spot checked and it looked reasonable. (the script took about 10 minutes to run against a remote SVN. But I had almost 400 tags so that’s not bad.)

The script is long so I included it at the bottom of the post.I put in a file named generateSvnGitMapping.sh

Step 3 – Actually create the Git tags

I ran another script I wrote to actually create the github tags. We had one SVN tag with a single quote in it. This tag was created in error. Since it was recreated with the proper name, I choose not to migrate it. I put this script in a file named generateTags.sh: 

for line in `cat ../tagMapping-oneLine.txt`
do
echo $line
echo ${line} | awk -F',' '{ print git tag -a $1 -m"re-creating tag for SVN @$2 commit" $3 }'
done

Then I

  1. ran generateTags.sh > tags.sh
  2. Placed script in git repo directory: mv tags.sh new-git-repo
  3. cd new-git-repo
  4. Ran script ./tags.sh
  5. Delete script rm tags.sh
  6. Listed tags to ensure seemed reasonable: git tags -l

Step 4 – Push tags to GitLab (or GitHub)

git push --tags

The script for step 2

#!/bin/bash

# To run, pass two parameters: 
# 1) The location of the local git repository
# 2) The file containing the tag names you'd like to check
# 3) The URL of the base SVN (not including /tags)
# ------------------------------------------------

# gets svn revision number for a svn tag
# $1 = svn url (ex: https://svn.com/svn)
# $2 = tag (ex: my_tag)
# sets variable svn_revision
set_svn_revision() { 
  svn_revision=`svn log "$1/tags/$2"/ --limit 1 | 
       head -2 | tail -1  | awk '{print $1}'`

   # remove leading "r" so just returns the number
   svn_revision=${svn_revision#r}
}

# ------------------------------------------------

# sets the git commit number for a svn revision
# if no commits are found (or more than one comment has this tag), exits the program
# $1 = svn commit (ex: r1234)
# $2 = tag name (ex: my_tag)
# sets variable git_commit
set_git_commit() {
 
  # since didn't migrate tags, need commit right before the tag
  # this might be a few commits back in some cases so check 5
  max_to_try=5
  num_attempts=0
  commit_right_before_tag=""
  previous_commit=$1
  
  while [ "$commit_right_before_tag" == "" ];
  do
    # add character before/after in git-svn log message to avoid ambiguity
    candidate_commit="@$previous_commit "

    num_lines=`git log --grep "$candidate_commit" | grep commit | wc -l`

    # if found a commit, use it. otherwise try some more
    if [ "$num_lines" -ne "0" ]; then
      commit_right_before_tag=$candidate_commit
    fi 
    
    num_attempts=$(($num_attempts-1))
    previous_commit=$(($previous_commit-1))
    
    # if  tried too many times, give up
    if [ "$num_attempts" -gt "$max_to_try" ]; then
      echo "commit ($1) for tag name ($2) not found. Aborting. Please check git repo or remove this tag from the input"
    exit
    fi 

  done

  git_commit=`git log --grep "$commit_right_before_tag" | head -1 | awk '{print $2}'`
}

# ------------------------------------------------

if [[ $# -ne 3 ]]
then 
    echo "Requires three parameters:"
    echo "1) Path to git repo"
    echo "2) name of file containing tags"
    echo "3) url to svn"
    exit
fi

git_dir="$1"
tag_names_file="$2"
svn_url="$3"

all_tag_names=`cat $tag_names_file`

current_dir=`pwd`
cd "$git_dir"

for tag_name in `echo $all_tag_names`
do
    tag_name_without_trailing_slash=${tag_name%/}
    set_svn_revision "$svn_url" "$tag_name"
    set_git_commit "$svn_revision" "$tag_name"
    echo "git tag -a "$tag_name_without_trailing_slash" -m \"re-creating tag for SVN @$svn_revision commit\" $git_commit"
done

cd $current_dir

how not to migrate from subversion to git

Posted on by
Reply

You know how you typically read blog posts of what to do that works. And not all the things people tried that didn’t work. This post is dedicated to what didn’t work.

Also see:

Don’t do this #1 – Migrate from a remote repository

Migrating from SVN to Git requires a large number of network roundtrips (for a large repository.) This slows things down greatly. It’s better to export/dump the repository and run everything locally.

See the main blog post for how to create a local dump/rep

Don’t do this #2 – Split the dump by project

I had the idea to split the SVN full dump file into smaller SVN dump files by project. I chose to preserve revision numbers and not use “renumber-revs”. We used the revision numbers in our release notes. Here’s a sample command:

svndumpfilter include "IntegrationTests" --drop-empty-revs < full.dmp 
  > project_IntegrationTests.dmp

We had one project that consists of the majority of the SVN code base (the forum software.) All of the tags were for this project. I thought to import this one as “full.dmp” and just delete the “trunk” projects afterwards for this one. That way I’d only be filtering the smaller/safer ones.

None of this was necessary! You can just point migration at the same full SNV dump with different paths to migrate projects into their own repositories.

Don’t do this #3 – Check out the entire repository including tags

Migrating using “git svn clone” requires an authors.txt to map SVN users to GitHub names/emails. I had the idea to check out the entire repository including tags and running svn log on it to get the committers. After 90 minutes, I gave up on this idea.

Don’t do this #4 – Assume that all authors/committers are people

There were a couple commits from Jenkins which seems reasonable. There were also a couple commits as “root”, “test” and other random users. Looking at the readme.txt from one of those commits, it looks like a command line import.

Don’t do this #5 – Guess at what should be in the authors.txt file

We have about 90 users in our authors.txt file. I thought I would save time by only putting the people I thought were committers in the authors.txt. This was a problem for a few reasons:

  • About 30 people committed to the main project
  • A few people committed who no longer have access to the code base.
  • We had some “funky” committers including “root” and “test”

This meant I kept running the “git svn clone” command, having it fail on missing users, adding them to authors.txt and resuming the run (re-running automatically resumes).

It would have better to us svn log on trunk to get all the authors or the –authors-prog flag to specify a command to fill in any defaults. This would have let me write “Unknown” for the funky ones and be done with it.

Don’t do this #6 – Make assumptions about project structure

At the top level, the repository had:

  • about 20 projects (directly at the root level, not under trunk)
  • a branches directory
  • a tags directory

I foolishly assumed that meant that the 20 projects had the code directly inside them. And sometimes that was true. However, for about 5 projects, there was a nested trunk/branches/tags structure under that project.

We all know that thing about standards. There are so many….

Don’t do this #7 – Migrate 300 large tags

This project uses Ant (and not Ivy) so there are a lot of jar files in the repository. This means tags are large. With just under ten thousand commits and just under 400 tags, this proved to be just too much.

Watching the “git svn clone” procedure, it goes through commit 1-n as it goes. This means the later commits/tags need to go through a large amount of work to make progress. Despite that, it was surprisingly linear.

After 12 hours, it had migrated 2700 commits and after 26 hours, it was up to commit 5446. At the 18 hour mark, it was up to commit 6926. (At the 24 hour mark, I decided to abandon this approach. I let it run until I needed to shut down my computer to see what would happen.)

Most of the wasted time was for the tags. Which in SVN are a copy. In Git, they are just a label so this is a lot of unnecessary duplication in a migration.

See another approach for migrating tags