Want to show your appreciation? Please to my charity.

Monday, December 21, 2015

Downsize HDD on Your Windows Box

Upgrade a laptop from HDD to SSD is probably the best investment for performance. I have done it for a lot of them for my friends and relatives. For most of time, it is always a good idea to take the opportunity to clean up the system by reinstalling the OS. But today, my friend handed me a 2nd hand laptop with Windows 7 and Office 2010 on it but there is neither COA in the back of the computer, nor does she have the product keys.

So I was challenged to downsize her 320GB HDD to 128G SSD. I started the venture and finally done now although there were quite some bumpers along the way. I will not go through every each mistakes that I have made but thought should document the correct process for someone else or myself will need to do the same again.

Backup Your Computer

Use an external HDD or NAS to back up your computer, Windows 7 have build-in support to do this. Go to Control Panel and search for “Back and Restore”.

Preparation

  1. You can either use DVD Writer drive or USB drive if your computer can boot from USB HDD. I used two microSD card with a tiny USB card reader. I have a batch of 4GB/8GB microSD cards but essentially one used 4GB and another used 2GB. You may be able to fix all into one 4GB or 8GB card.
  2. You’ll need to somehow attach both disk to the computer. Either your computer can fit both or use an external HDD enclosure. This was easy for me because the SSD was mSATA so Thinkpad X220 can fit in both HDD and SSD.
  3. Windows 7 installation DVD, or bootable USB Drive. I have Windows 7 installer in one of microSD cards. To create one, you’ll need a 4GB+ USB memory stick, or any memory card with an USB card reader, and use the tools below
    • ISO to USB was the tool I used long time back to create a bootable USB memory stick from an ISO image
    • If you don’t have Windows 7 installation DVD ISO image, you maybe able to download one somewhere or create on from the installation DVD, I have used both Win32 Disk Imager (just for that) and DAEMON Tools Lite (plus a virtual drive) with great success.
    • You can also mix this with next two items for this purpose, but I already and also prefer to have a separate one for Windows Installer.
  4. GParted for resizing partition. The site has instruction to created a dedicated CD or USB. It also come with most of Linux DVD. I used my existing Ubuntu ISO.
  5. Clonezilla for disk cloning. Similarly, the site has instruction to created a dedicated CD or USB. But I used Easy2Boot to combine it with Ubuntu image into one bootable USB. If you don’t have and need Ubuntu, you can just combine GParted and Clonezilla with Easy2Boot. You can possibly further combine Windows 7 Installer into the same Easy2Boot disk if you’d like.

In Action

  1. Again, backup your computer. You can backup system after the clean up in next step but you risk the mistake you may take during the clean up.
  2. Clean up your existing HDD to make sure the C drive is much less than the size of new SSD so you have enough free space after downsizing. This was not a problem for me as existing HDD has only 45GB data in it. Having larger free size can also help with defragmentation and resize later. Sometime, you just need to move videos and images to external storage first.
  3. Defragment the C drive. This will take a long while.
  4. Attached new SSD drive to the computer, initialize the disk with MGR. I don’t have much experience with GPT and my the existing disk was MBR too.
  5. Boot into GParted, resize the C partition (not System Reserved partition that is only 100M) on HDD to be able to fit into SSD. You can check the SSD size in GParted, I resized C on HDD to be 1GB less than SSD capacity.
  6. Reboot computer normally into Windows 7, it will most likely run check disk on C, or even run system recovery. Let it complete what it need to do. You should have a much smaller C drive now.
  7. Reboot into Clonezilla, go to command line and run ‘sudo fdisk’. First record the exact parameter of the 100M system reserved partition. Exit and fdisk the SSD drive. Create first partition with exactly same parameter as we recorded for system reserved partition. Create another partition for the remaining space. Make system reserved partition active and make sure to change ID to 7 for both partitions. Exit command line and back to Clonezilla
  8. Use local partition to partition clone option to copy the 100MB system reserved partition first, then use the same process to copy C partition from HDD to SDD.
  9. Remove the old HDD now and try to boot from SSD. If everything get up and running and you have C drive, Congrats! But
  10. But I got an error screen in text immediately attempting to boot from SSD. Need to boot from Windows installer DVD or USB and run repair. This should allow you to boot into Windows.
  11. In my case, it took a long time to log in to an empty screen. Press Ctl-Alt-Delete and select Start Task Manager. Use File->New Task (Run…) menu item and click on Browse button. I noticed that the C driver was missing and my primary partition as E drive, yours may be a different drive. Please remember the driver letter as we’ll need it later. You can start explore.exe or cmd.exe from there but what you can do with the computer is greatly limited. the real problem is that you cannot start regedit.exe.
  12. Fix the driver letter problem! Boot with your Windows Installer DVD or USB again into recovery console, follow the instructions here, but you need to open SYSTEM hive instead, give name as NEWSYSTEM
  13. You need to edit the registry key to change the drive letter of your primary partition. Replace HKEY_LOCAL_MACHINE\SYSTEM\ in the instruction with HKEY_LOCAL_MACHINE\NEWSYSTEM\
  14. Restart the computer and now all is good. I add the old HDD back and re-partitioned to be a large spare drive.

Saturday, May 30, 2015

Subversion (SVN) To Git Migration

Summary

There are many blogs and pages tell you how to use git-svn to do the work. But I found many of them works for simple use case but becomes cumbersome to use for complex, none-standard Subversion repository layout. This document try to provide a comprehensive guidance to make your migration from Subversion to Git a stress free process.

Repository Layout

Your repository layout has a deep impact to the effort of migration. Most of Subversion repository fall into one of below three layout categories.

Standard Layout

Standard layout repository strictly follows the trunk, branches and tags naming convention with exact case-sensitive top level directory names. There should be no other top level directory that you care about except those three.
WARNING: If you need to import any other top level directory as a branch, then it is no longer a standard layout so you need to follow instructions in None-Standard Layout section.
WARNING: Some of repositories use title case convention. i.e. with top level directory of Trunk, Branches and Tags. Those repositories are NOT standard layout due to the upper case letters. They are categorized as Semi-Standard Layout.
Note: For the purpose of this discussion the naming convention of individual branch or tag is irrelevant. Although in strict speaking, one would expect individual branch is named as major.minor (e.g. 1.2, 1.3) and tag is named as major.minor.revision (e.g. 1.2.0, 1.3.4).
To migrate Subversion repository with Standard Layout, follow the instructions in Using Stash Import Utility if you use Atlassian Stash as your GIT repository manager, or you can always use svn2git for any git repository.

Standard Layout repository example.

project-one
    +-- branches
            +-- branchname1
            +-- Branchname2
            +-- ...
    +-- tags
            +-- tagName1
            +-- tagname2
            +-- ...
    +-- trunk
Note: Repositories don't have all three top level directories still conform to standard layout as long as other top level directories conform to the naming convention. For example below repository doesn't have tag but it is considered as Standard Layout.
some-service
    +-- branches
    +-- trunk

Semi-Standard Layout

Semi-Standard Layout has similar directory structure as Standard Layout but doesn't follow the strict top level directory naming convention. It has similar trunk, branches and tags concept the top level directories are named differently, including some use title case, some use singular form, or some are completely different. There should be no other top level directory that you care about except those three.
WARNING: If you need to import any other top level directory as a branch, then it is no longer a standard layout so you need to follow instructions in None-Standard Layout section.
To migrate Subversion repository with Semi-Standard Layout, follow the instructions in Using Stash Import Utility if you use Atlassian Stash as your GIT repository manager, or you can always use svn2git for any git repository.

Below are some examples of Semi-Standard Layout.

Example 1: use title case Trunk, Branches and Tags.
project-two
    +-- Branches
            +-- branchname1
            +-- Branchname2
            +-- ...
    +-- Tags
            +-- tagName1
            +-- tagname2
            +-- ...
    +-- Trunk

Example 2: use singular trunk, branch and tag.
repo1
    +-- branch
            +-- branchname1
            +-- Branchname2
            +-- ...
    +-- tag
            +-- tagName1
            +-- tagname2
            +-- ...
    +-- trunk

Example 3: use main, features and releases instead of trunk, branches and tags.
STOP! Some repositories store binary artifacts under Releases directory, that is fundamentally different and we should never import any release binary into Git.
repo2
    +-- features
            +-- branchname1
            +-- Branchname2
            +-- ...
    +-- releases
            +-- tagName1
            +-- tagname2
            +-- ...
    +-- main
Note: Repositories don't have all three top level directories still conform to Semi-Standard Layout as long as other part of directory structure conforms to it. For example below repository doesn't have tag but it is considered as Semi-Standard Layout.
some-service
    +-- features
    +-- main

None-Standard Layout

None-Standard Layout refer to repositories that is not in Standard or Semi-Standard layout. Their branches are all reside under the root of project directory. For example:

project-three
    +-- Current
    +-- Dev
    +-- QA
    +-- UAT
    +-- Release (binary)
You'll need to use svn2git to migrate such repositories to Git.

Using Stash Import Utility

Stash has a build in utility to import Subversion repository as part of the repository creation wizard or thought the setting page of the repository. This can be used to import Subversion repositories with Standard and Semi-Standard Layout.
STOP! if you haven't read about the repository layout.
STOP! if you need resync from Subversion to Git later. Please consider using svn2git.
1. Create a new repository
  • If you already have the repository created and still empty, move on to next step.
  • If you need to reuse a repository that is not empty, sorry you cannot use this utility. Try to use svn2git.
Follow the instructions below to create a new Git repository in Stash.
  1. Go to your Stash web interface and login.
  2. Select a Stash project that is appropriate for the repository that you are creating.
  3. Click "Create Repository" button next to the Stash project name located in the upper-left corner of the project page. If you don't see the button, and you already signed in to Stash, then you need to request for permission.
  4. Enter the name of the repository, click on the "Create repository" button.
You should get an confirmation page telling you that an empty repository is now created. In the bottom of the page, you can find a button labeled as "Import from SVN", check on that to start importing. Move on to step No.3.  

2. Reusing an empty repository
If you have an empty repository that you want to use as target of migration. Go to the project and click on settings tab, if you don't see settings tab, you need to ask for permission to get there. In the settings page, click on the 'Import from SVN' link at the end of left menu to start importing.

3. Import repository
  • Enter the URL to the subversion repository that you are importing.
  • For Standard Layout project, you can leave Trunk, Branches and Tags field as default. Otherwise, enter the proper value for your repository. You can leave it the default if you don't have branches and/or tags.
  • Enter your username and password to access SVN repository.
Below table listed three examples:
Layout Trunk Branches Tags Comments
Standard trunk branches/* tags/* Leaves default value in place
Semi-Standard Trunk Branches/* Tags/* Enter the values for Trunk, Branches and Tags.
Semi-Standard Current branches/* tags/* For single branch import, enter the values for Trunk but leaving Branches and Tags as default.
Click "Import" button. Stash first start to gather the author list, this may take a few seconds to a few minutes and it will return back with a message:
Default authors mapping created. Review it, adjust if necessary and proceed with the import.

4. Review author mapping
Before Stash actually import, it gather author information and ask you to review author mapping. Click on "Continue" button and you'll get a popup windows for you to review the author mapping. Make necessary changes and click on "Continue" button on the popup.
Stash will start to import the repository and update the progress on the page. You can safely close the browser now. You will receive an email telling you that the import is in progress and receive another email when it is done.

Congratulations! Now you have completed the migration from Subversion to Git.

Using svn2git

svn2git is a great tool to migration from Subversion to Git. It uses git-svn under the hood but does a much better, cleaner job. svn2git is a better choice unless you are a guru of both Subversion and Git and a master of svn-git.

Install svn2git

Below lists the installation for various platform but your mileage may vary. If you get it running on a platform not listed below successfully, please update this wiki.
Mac/Linux
 sudo gem install svn2git
Windows (Command Prompto)
 gem install svn2git

Run svn2git

You always run svn2git in an empty directory. After executing svn2git, the directory becomes the root of the converted Git repository that mirrors the Subversion repository.
Note: If you executed svn2git or git-svn with wrong parameter and you want to start over again, you must delete entire directory first then recreate the directory, otherwise you'll get weird error.
WARNING: svn2git can take very long time for large repository, make sure to run on a machine that is close (low network latency) to the Subversion repository server.


Standard Layout Repository

To convert a Standard Layout repository, all you need to do is to run it with URL to Subversion repository. e.g.
mkdir project-one
cd project-one
svn2git https://svn.somecompany.com/svn/project-one/

Semi-Standard Layout Repository

Semi-Standard Layout repository has the same directory structure as Standard Layout repository. The difference is the directory name of trunk, branches and tags. You use corresponding option parameter to specify them. If you have no branch or tag, you can omit it.

Example 1
mkdir project-two
cd project-two
svn2git https://svn.somecompany.com/svn/project-two/ --trunk Trunk --branches Branches --tags Tags

Example 2
mkdir project-simple
cd project-simple
svn2git https://svn.somecompany.com/svn/project-simple/ --trunk Current

None-Standard Layout Repository

For None-Standard Layout repositories, the difficult part is the branches reside at the top level directory. Let's ignore them as the first step and run svn2git with parameters that best matches the Standard or Semi-Standard Layout.

Using project-three as an example,
project-three
    +-- Current
    +-- DEV
    +-- QA
    +-- UAT
    +-- Release (binary)

We will take DEV as trunk, take Current, UAT and QA as branches. Let's start with ignoring the top level branches and convert the DEV first.
 
mkdir project-three
cd project-three
git svn init --trunk DEV --prefix svn/ https://svn.somecompany.com/svn/project-three/

Wait for it to complete and then edit the .git/config file to add three lines to the svn-remote section for Current, UAT and QA.
WARNING: Make sure you have all branches well defined and there is no typo of the names, they are case-sensitive. If you missed one and want to include it later, you'll need to clean up the directory and start from scratch.
[svn-remote "svn"]
 url = https://svn.somecompany.com/svn/project-three
 fetch = DEV:refs/remotes/svn/trunk
 fetch = Current:refs/remotes/svn/Current
 fetch = UAT:refs/remotes/svn/UAT
 fetch = QA:refs/remotes/svn/QA
Note: You can also change the branch name in this process. For example, if you'd like to rename the Current branch in Subversion to Test in Git you can use
 fetch = Current:refs/remotes/svn/Test
Then run fetch and rebase. The fetch operation may take long time to complete if you have a big repository.
git svn fetch
svn2git --rebase

Push to Git Repository Server

If you don't already have a remote Git repository to push to, create one first. Then push all branches to the remote repository.
Find the URL to the remote Git repository, which is the same URL you used to clone remote repository. Run below commands in the directory of your local Git repository you just imported from Subversion, replace with the URL to the remote Git repository. 

git remote add origin <URL>
git push origin --all

Congratulations! Now you have completed the migration from Subversion to Git. You are safe to delete your local copy of the repository but if you need to resync from Subversion to Git again, you'll need your local repository.

Resync From Subversion to Git

You can start the migration when still allowing later changes in Subversion to be brought into Git. This is useful when you are moving an actively developed repository. Often you need to prepare the build scripts to use the new Git repository while allow development and build activity continue to occur in the old Subversion repository.
WARNING: Two way sync is neither tested nor recommended. We do not encourage to try have commits go into both old Subversion and new Git repository, then attempt to keep them in sync later. It will be difficult and risky if at all possible.
Go into the local Git repository that you have created and used for svn2git migration and run below commands

svn2git --rebase
git push origin --all

Q & A

Q: Can svn2git support author mapping?
A: Yes, see https://github.com/nirvdrum/svn2git#authors