Differential D2071

Add some tools to simplify the use of git
ClosedPublic
Actions

Authored by rstone on Mar 16 2015, 2:52 AM.

Details

Reviewers: None
Commits: rS307680: Add some tools to simplify the use of git

Summary

Add some scripts that wraps some FreeBSD Project infrastructure
and simplifies using them with git. The scripts are:

arcgit, which creates a series of reviews in Differential
importgit, which applies a series of git commits to svn

Diff Detail

Lint

No Lint Coverage

Unit

No Test Coverage

Build Status

Buildable 4726
Build 4780: arc lint + arc unit

Event Timeline

rstone updated this revision to Diff 4235.Mar 16 2015, 2:52 AM

rstone retitled this revision from to Add some tools to simplify the use of git.

rstone edited the test plan for this revision. (Show Details)

rstone updated this object.

Herald added a subscriber: imp. · View Herald TranscriptMar 16 2015, 2:52 AM

arcgit is a pretty handy script, thanks! It's slightly annoying that one has to remember to use "-r C1~..C2" instead of "-r C1..C2" for an inclusive range of commits, although I guess that's consistent with git commands. Maybe include the "~" in the usage message?

It'd also be useful to have some sort of semi-dry run mode. I just used it for the first time and wanted to try creating reviews without specifying any reviewers, but the interface doesn't allow that. I just commented out the section that adds them and manually added them in the browser after the reviews were created.

tools/tools/git/arcgit
44	This should be "-r commit1..".

Add -n option to arcgit to perform a dry run, and fix some typos.

rstone marked an inline comment as done.Aug 8 2016, 3:26 PM

Closed by commit rS307680: Add some tools to simplify the use of git (authored by rstone). · Explain WhyOct 20 2016, 6:24 PM

This revision was automatically updated to reflect the committed changes.

Revision Contents
Changeset List

Path

Size

tools/

README

1 line

git/

HOWTO

144 lines

arcgit

212 lines

importgit

162 lines

Diff 19122

View Options

tools/tools/README

	Show All 23 Lines
	ether_reflect An Ethernet packet reflector for low level testing.			ether_reflect An Ethernet packet reflector for low level testing.
	find-sb Scan a disk for possible filesystem superblocks.			find-sb Scan a disk for possible filesystem superblocks.
	gdb_regofs A simple tool that prints out a register offset table			gdb_regofs A simple tool that prints out a register offset table
	for mapping gdb(1) register numbers to struct reg and			for mapping gdb(1) register numbers to struct reg and
	struct fpreg offsets. The tool is useful on selected			struct fpreg offsets. The tool is useful on selected
	platforms only.			platforms only.
	genericize Turn a kernel config into something that can more easily			genericize Turn a kernel config into something that can more easily
	be diffed against the appropriate GENERIC.			be diffed against the appropriate GENERIC.
				git Tools to simplify the use of git by committers.
	hcomp Compress header files by removing comments and whitespace.			hcomp Compress header files by removing comments and whitespace.
	html-mv Rename HTML generated filenames to human readable filenames.			html-mv Rename HTML generated filenames to human readable filenames.
	ifinfo Uses the interface MIB to print out all the information			ifinfo Uses the interface MIB to print out all the information
	an interface exports in an ugly form.			an interface exports in an ugly form.
	iso Tool to compare the iso3166 and iso639 files in			iso Tool to compare the iso3166 and iso639 files in
	/usr/share/misc with the data from the master sites.			/usr/share/misc with the data from the master sites.
	iwi Tools specific to the Intel PRO/Wireless 2200BG/2225BG/2915ABG			iwi Tools specific to the Intel PRO/Wireless 2200BG/2225BG/2915ABG
	support.			support.
	Show All 30 Lines

View Options

tools/tools/git/HOWTO

This file was added.

				# $FreeBSD$

				This directory contains tools intended to help committers use git when
				interacting with standard FreeBSD project resources like Differential or svn.

				I. arcgit

				arcgit is a wrapper script around the arc command line tool that simplifies the
				automatic creation of a series of Differential reviews from a series of git
				commits. The intended workflow is:

				1. Create a series of commits in git. Each commit will be a separate review, so
				try to make each commit as self-contained as possible. The series of commits
				should demonstrate a logical progression towards your end goal. For example,
				one commit might refactor existing code to expose a new API without changing
				any current functionality. A subsequent commit could then introduce your new
				code that uses the new API.

				It usually will not be helpful to present your code in the order in which it
				was actually written and can actually be harmful. For example, if you
				introduced a bug early in your development process that you fixed in a
				subsequent commit, it is a waste of your reviewer's time to have them review
				old code with known bugs. Instead, it would probably be best to squash the
				bug fix into the commit that introduced it, so that the bug is never
				presented to your reviewers in any review.

				The commit headline and commit message will be imported verbatim into
				Differential, so try to give each commit a meaningful commit message that
				gives your reviewers the necessary context to understand your change.

				2. Create your reviews bu running this command in your git repo:
				$ arcgit -r C1~..C2 -R reviewer -T testplan

				C1 should be the first commit that you want reviewed, and C2 should be the
				last commit that you want reviewed. You may add multiple reviewers by
				specifying the -R option multiple times. You can CC (AKA subscribe) people
				to a review with the -C option. Note that if you subscribe a mailing list
				to a review, the mailing list will be emailed for every comment or change
				made to each review. Please be judicious when subscibing mailing lists to
				reviews. It may be better to instead send a single email to the appropriate
				list announcing all of the reviews and giving a short summary of the change
				as a whole, along with a link to the individual reviews.

				3. When you need to make a change and upload it to a review, use the following
				process. First, check out the branch corresponding to the review (arcgit
				automatically creates this branch for every review that it creates):

				$ git checkout review_D1234

				Next, make your change and perform whatever testing is necessary. Commit it
				to your repository with this command:

				$ git commit --fixup HEAD

				You can upload the change to the code review by running this command in your
				repository while (ensure that you are still on the review_D1234 branch):

				$ arc diff --update D1234 review_D1234_base

				When you run arc, it will pull up your editor and give you a chance to
				change the message that will be shown in Differential for this upload. It's
				recommended that you change it from the default "fixup! ...." as that does
				not give your reviewers an indication of what changes were made. It's not
				recommended that you give the code review fixes meaningful commit messages
				directly because we will be using git's autosquash feature in the next step,
				which depends on the fixup! tag being present.

				Do not merge in other branches, or rebase your review branches at this point.
				Any changes that are made will show up in your reviews, and that will create
				extra noise that your reviewers will have to ignore. If a reviewer requests
				a change that requires your commit to be based off of a later version of
				head, I would suggest deferring the change from the review and creating a
				new review with the change once you hit step 5.

				4. Once the reviews have been approved, you need to prepare your patch series
				to be committed. This involves squashing the fixes made in code review
				back into the original commit that they applied to. This gives you a clean
				series of commits that are ready to be commited back to svn.

				First, merge each of your review branches back into your main development
				branch. For example:

				$ git checkout myfeature_dev
				$ for branch in review_D1234 review_D1235 review_D1236; do \
				git merge $branch \|\| git mergetool -y \|\| break; done

				Next, do an interactive rebase to squash your code review fixes back into the
				main commit:

				$ git rebase -i -k review_D1234_base

				review_D1234 should be the name of the first review that was created for
				you in step 2. For every commit, your editor will be pulled up and you will
				be given one last chance to edit your commit message. Make sure that you fill
				in the "Reviewed by:" tag indicating who accepted the review. This would
				be a good point to add other tags like MFC after:, Sponsored by:, etc.

				The rebase will not introduce any actual changes to your code if done
				properly. You can use this command to double-check that no changes were
				inadvertently made here:

				$ git diff myfeature_dev@{1}

				5. Finally, you should get up to date with the latest changes from head:

				$ git pull --rebase origin master

				It is not recommended that you combine this step with the rebase done in the
				previous step. The reason for this is that if you perform an interactive
				rebase that changes the commit that you branch from, it becomes much more
				difficult to use the reflog to confirm that the interactive rebase did not
				introduce unwanted changes.

				At this point, you are ready to commit your changes to head. The importgit
				script can be used to import your commits directly into git.

				II. importgit

				importgit is a script that can take a series of commits from git and commit them
				to a svn repository. The script uses the git commit messages for the svn commit
				message, which allows importgit to be fully automated. This does mean that once
				you start importgit, it will start commit things to svn without giving any
				further chance to sanity check what it's doing.

				importgit only supports importing commits that add or modify files. It does not
				support importing commits that rename or delete files, to ensure that git's
				rename detection heuristics do not introduce an error in the import process.
				importgit also does not support importing merge commits. Only linear history
				can be imported into svn.

				importgit must be run from a clean subversion checkout. You should ensure that
				the working tree is up-to-date with "svn up" before running importgit.
				importgit will run svn directly, so make sure that your ssh-agent is running
				and has your ssh key loaded already. Run importgit as follows:

				$ importgit -r D1~..D2 -g /path/to/git/repo

				This will import every commit between D1 and D2, including both D1 and D2. The
				invocation is very similar to the invocation given to arcgit but there is an
				important point to note. When you rebased your commits as you followed steps 4
				and 5, the commit hashes of all of your commits changed, including C1 and C2.
				You must go back and find the new commit hashes of your commits to pass to
				importgit. Passing -r C1~..C2 would import your commits as they were before
				your code review fixes were applied.

View Options

tools/tools/git/arcgit

This file was added.

Property	Old Value	New Value
File Mode	null	100755

				#!/bin/sh
				#
				# Copyright (c) 2015 Ryan Stone. All rights reserved.
				#
				# Redistribution and use in source and binary forms, with or without
				# modification, are permitted provided that the following conditions
				# are met:
				# 1. Redistributions of source code must retain the above copyright
				# notice, this list of conditions and the following disclaimer.
				# 2. Redistributions in binary form must reproduce the above copyright
				# notice, this list of conditions and the following disclaimer in the
				# documentation and/or other materials provided with the distribution.
				#
				# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
				# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
				# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
				# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
				# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
				# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
				# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
				# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
				# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
				# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
				# SUCH DAMAGE.
				#
				# $FreeBSD$

				# This script is used to submit a series of git commits to Differential. Each
				# commit is submitted as a separate review. For each review, this script will
				# create a branch called review_DXXXX (e.g. review_D2185 for Differential
				# revision D2185). When you need to make a change to a review, checkout the
				# review_D2185 branch, commit your change with "git commit --fixup HEAD". To\
				# upload the change to Differential, use the command:
				# $ arc diff --update D2185 review_D2185_base
				#
				# When your reviews are complete, merge all of the review_DXXXX branches
				# together, and then do a git rebase -ik to meld the code review fixes into the
				# commit that they fixed. Now you have a clean series of patches to commit to
				# svn.

				usage()
				{
				echo "arcgit < -c commit \| -r commit1~..commit2 > [ -R reviewer ] " >&2
				echo " [ -C subscriber ] [ -T testplan ] [-n]" >&2
				markjUnsubmitted Done Inline Actions This should be "-r commit1..". markj: This should be "-r commit1..".
				}

				error()
				{
				echo "$@" >&2
				usage
				rm -f $phab_before $phab_after $arc_msg
				exit 1
				}

				create_review()
				{
				local commit phab_id arc_dir
				unset phab_before phab_after arc_msg
				commit=$1

				phab_before=`mktemp -t arcoutput`
				phab_after=`mktemp -t arcoutput`
				echo "Create review for '`git show $commit -s --oneline`'"

				if [ -n "$dry_run" ]
				then
				return
				fi

				git checkout $commit > /dev/null \|\| error "Could not checkout $commit"

				arc_dir="$(git rev-parse --show-toplevel)/.git/arc"
				arc_msg="$arc_dir/create-message"
				mkdir -p $arc_dir
				git show -s --format='%B' HEAD > $arc_msg
				echo >> $arc_msg
				echo "Test Plan:" >> $arc_msg
				cat $test_plan >> $arc_msg
				echo >> $arc_msg
				echo "Reviewers:" >> $arc_msg
				echo "$reviewers" >> $arc_msg
				echo >> $arc_msg
				echo "Subscribers:" >> $arc_msg
				echo "$cc_list" >> $arc_msg
				echo >> $arc_msg

				arc list > $phab_before
				yes \| env EDITOR=true arc diff --create --allow-untracked HEAD~
				arc list > $phab_after

				headline="$(git show $commit -s --format=%s)"
				phab_id=`comm -13 "$phab_before" "$phab_after" \| fgrep "$headline" \
				\| egrep -o 'D[0-9]+:' \| tr -d ':'`

				if [ -z "$phab_id" ]
				then
				error "Could not get review ID"
				fi

				git branch review_${phab_id}_base HEAD~

				git checkout -b review_$phab_id
				cat - <<EOF \| git commit --allow-empty -F -
				squash! $headline

				Differential Revision: https://reviews.freebsd.org/$phab_id
				Reviewed by:
				EOF
				}

				unset range test_plan reviewers cc_list dry_run

				while getopts "c:C:nr:R:T:" o
				do
				case "$o" in
				c)
				range="${OPTARG}~..${OPTARG}"
				;;
				C)
				if [ -z "$cc_list" ]
				then
				cc_list="$OPTARG"
				else
				cc_list="$cc_list, $OPTARG"
				fi
				;;
				n)
				dry_run=1
				;;
				r)
				range=$OPTARG
				;;
				R)
				if [ -z "$reviewers" ]
				then
				reviewers="$OPTARG"
				else
				reviewers="$reviewers, $OPTARG"
				fi
				;;
				T)
				test_plan=$OPTARG
				;;
				esac
				done

				shift $((OPTIND - 1))
				OPTIND=1

				if [ -n "$1" ]
				then
				error "Unrecognized argument $1"
				fi

				if [ -z "$range" ]
				then
				error "-c or -r argument is mandatory"
				fi

				if [ -n "$test_plan" -a ! -r "$test_plan" ]
				then
				error "$test_plan is not readable"
				fi

				if ! type git > /dev/null 2> /dev/null
				then
				error "Install devel/git first"
				fi

				if ! type arc > /dev/null 2> /dev/null
				then
				error "Install devel/arcanist first"
				fi

				git update-index -q --refresh
				if ! git diff-index --quiet --cached HEAD
				then
				error "index is unclean"
				fi

				if ! git diff-files --quiet
				then
				error "Working directory is unclean"
				fi

				if git ls-files --other --error-unmatch . > /dev/null 2> /dev/null
				then
				error "Working directory contains untracked files"
				fi

				# We have to do a git checkout in order to run arc, so save the original branch
				# so that we can check it out again once we're done.
				if ! orig_branch=$(git symbolic-ref --short -q HEAD)
				then
				orig_branch=$(git show -s --pretty='%H' HEAD)
				fi

				git log --format=%H $range \| tail -r \| while read -r commit
				do
				create_review $commit < /dev/null
				done

				# Note that due to the use of the pipeline above, the body of the while loop
				# above runs in a subshell. If it exits with an error, execution resumes
				# here rather than exiting the script, so we have to cache the right exit code
				# and return it when we're done cleaning up.
				code=$?

				git checkout $orig_branch

				exit $code

View Options

tools/tools/git/importgit

This file was added.

Property	Old Value	New Value
File Mode	null	100755

				#!/bin/sh
				#
				# Copyright (c) 2015 Ryan Stone. All rights reserved.
				#
				# Redistribution and use in source and binary forms, with or without
				# modification, are permitted provided that the following conditions
				# are met:
				# 1. Redistributions of source code must retain the above copyright
				# notice, this list of conditions and the following disclaimer.
				# 2. Redistributions in binary form must reproduce the above copyright
				# notice, this list of conditions and the following disclaimer in the
				# documentation and/or other materials provided with the distribution.
				#
				# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
				# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
				# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
				# ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR OR CONTRIBUTORS BE LIABLE
				# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
				# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
				# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
				# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
				# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
				# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
				# SUCH DAMAGE.
				#
				# $FreeBSD$

				usage()
				{
				echo "importgit < -c commit \| -r c1..c2 > -g /path/to/git/repo" >&2
				}

				error()
				{
				local print_usage

				if [ "$1" = "-u" ]
				then
				shift
				print_usage=1
				else
				print_usage=
				fi

				echo "$@" >&2
				if [ -z "$print_usage" ]
				then
				usage
				fi
				exit 1
				}

				unset git range commit

				while getopts "c:g:r:" o
				do
				case "$o" in
				c)
				range="${OPTARG}~..${OPTARG}"
				;;
				g)
				git_repo=$OPTARG
				;;
				r)
				range=$OPTARG
				;;
				esac
				done

				shift $((OPTIND - 1))
				OPTIND=1

				if [ -n "$1" ]
				then
				error -u "Unrecognized argument $1"
				fi

				if [ -z "$range" ]
				then
				error -u "-c or -r argument is mandatory"
				fi

				if [ -z "$git_repo" ]
				then
				error -u "-g <repo> argument is mandatory"
				fi

				git="$git_repo/.git"

				if [ ! -d "$git" ]
				then
				error "$git_repo does not seem to be a git repo"
				fi

				if ! type git > /dev/null 2> /dev/null
				then
				error "Install devel/git first"
				fi

				if ! type svn > /dev/null 2> /dev/null
				then
				error "Install devel/subversion first"
				fi

				if [ -n "$(svn status)" ]
				then
				error "Working tree is not clean"
				fi

				git --git-dir=$git log --format=%H $range \| tail -r \| while read -r commit
				do
				echo "Applying `git --git-dir=$git show -s --oneline $commit`"

				if [ -n "$(git --git-dir=$git show --diff-filter=CDRTUXB $commit)" ]
				then
				error "Commit performed unsupported change (e.g. delete/rename)"
				fi

				if [ "$(git --git-dir=$git show -s --format=%P $commit \| wc -w)" -ne 1 ]
				then
				error "Cannot import merge commits"
				fi

				git --git-dir=$git diff --diff-filter=A --name-only \
				${commit}~..$commit \| while read -r newfile
				do
				if [ -f "$newfile" ]
				then
				error "New file $newfile already exists in tree"
				fi
				done

				# The previous while loop ran in a subshell, so we have to check if it
				# exited with an error and bail out if so.
				ret=$?
				if [ "$ret" -ne 0 ]
				then
				exit $ret
				fi

				git --git-dir=$git show $commit \| patch -p 1 -s \|\| \
				error "Failed to apply patch"

				git --git-dir=$git diff --diff-filter=A --name-only \
				${commit}~..$commit \| while read -r newfile
				do
				svn add --parents --depth=infinity $newfile \|\| \
				error "Failed to add new file"
				done

				# The previous while loop ran in a subshell, so we have to check if it
				# exited with an error and bail out if so.
				ret=$?
				if [ "$ret" -ne 0 ]
				then
				exit $ret
				fi

				git --git-dir=$git show -s --format='%B' $commit \| svn commit -F - \|\| \
				error "Failed to commit"
				done

Add some tools to simplify the use of gitClosedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 19122

tools/tools/README

tools/tools/git/HOWTO

tools/tools/git/arcgit

tools/tools/git/importgit

Add some tools to simplify the use of git
ClosedPublic
Actions

Revision Contents
Changeset List