Archive for October, 2009

mq ~ Mercurial Queues

2009-10-27

I’ve been experimenting with Mercurial Queues with the idea of sharing
their mysteries with OpenSolaris developers so we can all integrate
multiple changesets and keep one fix per changeset.

Essentially mq allows you to take a snapshot of your changes and
create a patch file. You may then make additional changes to be
applied to the same patch file (qrefresh) or create a new patch (qnew)
These patches may then be removed (qpop) and reapplied (qpush)
at your leisure. So that’s the theory, lets take a look using an example:

Enabling mq

To enable mq on OpenSolaris simply add it to your extensions in
.hgrc: 

$ head -2 ~/.hgrc
[extensions]
mq=

Some data to work on.

For the following examples I’ve created a repository and added to it a
simple C ‘First program’ and a make(1) file to compile and
delete the program… Complete with some faults to fix. 

$ hg init qexample
$ cd qexample
$ echo 'int\nmain(int argc, char *argv[])\n{\n\tprintf("hello word");\n}'
$ 1> hi.c
$ echo src='hi.c\nbin=${src:%.c=%}\nall: ${bin}\nclobber:\n\trm ${bin}'
$ 1> Makefile
$ hg commit -m 'First program' -u beginner -A hi.c Makefile
$ cd ..

Clone repository and initialise for use with mq

Create a clone of qexample called qex1 and initialise it for mq
using qinit -c. 

$ hg clone qexample qex1
updating working directory
2 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ cd qex1
$ hg qinit -c

Fix Makefile clobber target

The Makefile has some issues, firstly the clobber target. 

$ make clobber hi
rm hi
rm: hi: No such file or directory
*** Error code 2
make: Fatal error: Command failed for target `clobber'

First Patch, Makefile

So here is our first issue, the clobber build fails as ‘rm’ failed and
causes make to exit. To address that we can simply add a -f to rm,
so lets do that for our first patch. 

$ hg qnew Makefile

The qnew command states we want to create a new patch, the argument
is the name of the patch. This action also creates a changeset, don’t
be concerned about that now. 

$ hg log
changeset:   1:d8ddc677fa77
tag:         qtip
tag:         Makefile
tag:         tip
tag:         qbase
user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
date:        Tue Oct 27 21:17:03 2009 +0000
summary:     [mq]: Makefile
changeset:   0:390c5f52f8e3
tag:         qparent
user:        beginner
date:        Tue Oct 27 21:17:02 2009 +0000
summary:     First program
$ sed 's/rm /rm -f /' Makefile > m
$ mv m Makefile
$ make clobber hi
rm -f hi
cc    -o hi hi.c
"hi.c", line 4: warning: implicit function declaration: printf
$ hg qrefresh

After applying the fix clobber target now works. However, now the
compiler’s giving us warnings! That’s a new issue, so I update the
Makefile change using qrefresh and start a new patch:

Second patch, remove warnings.

This time qnew is used with the -m option to add a description.
Note this description is also applied to the changeset, it can be
changed later if necessary: 

$ hg qnew -m 'remove implicit function delcaration warnings' fix1
$ (echo '#include ‹stdio.h›'; cat hi.c) > new.c
$ mv new.c hi.c
$ hg log -l1
changeset:   2:4ffb58a8196f
tag:         qtip
tag:         tip
tag:         fix1
user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
date:        Tue Oct 27 21:17:03 2009 +0000
summary:     remove implicit function declaration warnings
$
$ hg qdiff
diff -r d0b382d0a781 hi.c
--- a/hi.c	Tue Oct 27 21:17:03 2009 +0000
+++ b/hi.c	Tue Oct 27 21:17:03 2009 +0000
@@ -1,3 +1,4 @@
+#include ‹stdio.h›
int
main(int argc, char *argv[])
{
$ make hi
cc    -o hi hi.c
$ hg qrefresh

The include was added to hi.c to remove the warning, and again the
change updated using qrefresh.

Third patch, retrospective patching

Its likely that you’ll start making changes before remembering to
start a patch with qnew. For those occasions the -f option comes
to hand to place all un-committed changes in to the patch. Verified
below with qdiff: 

$ ./hi
hello word$ # Doh! No new line, and it was meant to say 'world'!
$ sed 's/hello word/Hello world!\\n/' hi.c > new.c
$ mv new.c hi.c
$ hg qnew -f -m 'Fix world' message
$ hg qdiff
diff -r 8a838c77402f hi.c
--- a/hi.c	Tue Oct 27 21:17:03 2009 +0000
+++ b/hi.c	Tue Oct 27 21:17:04 2009 +0000
@@ -2,5 +2,5 @@
int
main(int argc, char *argv[])
{
-	printf("hello word");
+	printf("Hello world!\n");
}

Listing patches with qseries

The qseries command shows us which patches have been applied, use the
-v option to display ‘A’ (applied) and ‘U’ (un-applied) flags: 

$ hg qseries -v
0 A Makefile
1 A fix1
2 A message
$ cat hi.c
#include ‹stdio.h›
int
main(int argc, char *argv[])
{
printf("Hello world!\n");
}
$

Traversing the patches to make other changes.

Lets assume we want to make another change to the Makefile, but
we want to keep just the one patch for changes to Makefile.
To accomplish this we pop the other two patches off. In the following
that’s done by naming the patch to pop to: 

$ hg qpop Makefile
now at: Makefile
$ hg qseries -v
0 A Makefile
1 U fix1
2 U message

Note now that the other two patches are un-applied.

With the just the original patch applied modify the Makefile, update
the patch using qrefresh and then re-apply all the other patches
using qpush -a. 

$(echo '# Makefile example';cat Makefile)>new
$ mv new Makefile
$ hg qrefresh
$ hg qdiff
diff -r 390c5f52f8e3 Makefile
--- a/Makefile	Tue Oct 27 21:17:02 2009 +0000
+++ b/Makefile	Tue Oct 27 21:17:04 2009 +0000
@@ -1,5 +1,6 @@
+# Makefile example
src=hi.c
bin=${src:%.c=%}
all: ${bin}
clobber:
-	rm ${bin}
+	rm -f ${bin}
$ hg qpush -a

Caution!

Some caution is needed when you want two patches that edit the
same file. Its doable but because underneath patch(1) is being used
you may need to manually merge in rejects.

Another gotcha on older versions of Mercurial is if you add a file in
a patch-queue. If the patch is popped off you may need to re-add it
(hg add). Lets see that in action by adding a new file which of
course requires adding to the Makefile too… This seems to be fixed
however in version 1.3.1: 

$ hg qpush -a
applying fix1
applying message
now at: message
$ hg qnew bye
$ sed 's/Hello world/Bye/' hi.c > bye.c
$ hg add bye.c
$ : Make changes to makefile...
$ hg qrefresh
$ hg qpop Makefile
now at: Makefile
$ sed 's/hi.c/hi.c bye.c/' Makefile >new
$ head -2 new
# Makefile example
src=hi.c bye.c
$ mv new Makefile
$ hg qrefresh
$ hg qpush -a
applying fix1
applying message
applying bye
now at: bye
$ hg log bye.c || hg add bye.c # NEED TO CONFIRM!
$ hg log bye.c
changeset:   4:cdb5958dcd87
tag:         qtip
tag:         tip
tag:         bye
user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
date:        Tue Oct 27 21:17:05 2009 +0000
summary:     imported patch bye
$ make
cc    -o hi hi.c
cc    -o bye bye.c
$

Updating your repository – pull and update.

It is most likely someone is going to have integrated (push) between
your original clone and push, and that you’d like to merge those changes.

So Firstly, for the example, add a changeset to our parent repo; 

$ hg clone ../qexample tmp
$ cd tmp
$ echo "README...  TBD" > README
$ hg add README
$ hg commit -u beginner -m 'Added README for support.'
$ hg push
$ cd ..
$ hg incoming -v
comparing with /export/home/sm26363/Mercurial/Mercurial/qexample
searching for changes
changeset:   1:2048a9e0068b
tag:         tip
user:        beginner
date:        Tue Oct 27 21:17:05 2009 +0000
files:       README
description:
Added README for support.
$

Now Hold onto your seats, this is a bit of roller-coaster ride!

Firstly, we save the series of patches using qsave, with some
essential options:

  • -c : Copy Patch directory
  • -n : Specify name of directory – By default it uses patches.N
    where N is next number in series until an non-existent directory
    is found within the root .hg directory.

  • -e : Empty the queue status file – The status file holds
    information about which patches are applied, so deleting it makes
    it look as though the patch queue is empty. The changesets are
    stripped of their special tags. 

$ hg qsave -e -c -n incoming
copy /export/home/sm26363/Mercurial/Mercurial/qex1/.hg/patches to /export/home/sm26363/Mercurial/Mercurial/qex1/.hg/incoming
$ hg glog --template '{rev}:{node|short} {author|user}\n{desc|firstline}\n\n'
@  5:db6e2d5b6aa2 Stacey
|  hg patches saved state
|
o  4:cdb5958dcd87 Stacey
|  imported patch bye
|
o  3:d20726d3cc4d Stacey
|  Fix world
|
o  2:63144e5ebe68 Stacey
|  remove implicit function declaration warnings
|
o  1:be53b09aed16 Stacey
|  [mq]: Makefile
|
o  0:390c5f52f8e3 beginner
First program

we’re now ready to pull the incoming changes over, DON’T use the -u
option. The update needs to be done separately: 

$ hg pull
pulling from /export/home/sm26363/Mercurial/Mercurial/qexample
searching for changes
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files (+1 heads)
(run 'hg heads' to see heads, 'hg merge' to merge)
$ hg glog --template "{rev}:{node|short} {author|user}\n{desc|firstline}\n\n"\
-l3
o  6:2048a9e0068b beginner
|  Added README for support.
|
| @  5:db6e2d5b6aa2 Stacey
| |  hg patches saved state
| |
| o  4:cdb5958dcd87 Stacey
| |  imported patch bye
| |

The pull done, as normal we have multiple heads and possibly files
that need merging. But as we’re using patches we’ll simply overwrite,
local changes using option -C and re-apply the patches using qpush
with the following arguments:

  • -m : Merge from another queue – This triggers a three-way merge
    if the patch fails to apply with the applicable changeset. The
    result is a new patch based on the changes.

  • -n : Name of other-queue – the backup from previous step.
  • -a : All patches. 

$ hg update -C tip
3 files updated, 0 files merged, 1 files removed, 0 files unresolved
$ hg qpush -m -n incoming -a
merging with queue at: /export/home/sm26363/Mercurial/Mercurial/qex1/.hg/incoming
applying Makefile
applying fix1
applying message
applying bye
now at: bye
$
$ hg glog --template "{rev}:{node|short} {author|user}\n{desc|firstline}\n\n"
@    11:8db2598210ab Stacey
|\   imported patch bye
| |
| o    10:2d966b806fb1 Stacey
| |\   Fix world
| | |
| | o    9:e63a43ac5532 Stacey
| | |\   remove implicit function declaration warnings
| | | |
| | | o    8:a1205da4671a Stacey
| | | |\   imported patch Makefile
| | | | |
| | | | o  7:3f838495d0e4 Stacey
| | | | |  [mq]: merge marker
| | | | |
| | | | o  6:2048a9e0068b beginner
| | | | |  Added README for support.
| | | | |
+---------o  5:db6e2d5b6aa2 Stacey
| | | | |    hg patches saved state
| | | | |
o | | | |  4:cdb5958dcd87 Stacey
|/ / / /   imported patch bye
| | | |
o | | |  3:d20726d3cc4d Stacey
|/ / /   Fix world
| | |
o | |  2:63144e5ebe68 Stacey
|/ /   remove implicit function declaration warnings
| |
o |  1:be53b09aed16 Stacey
|/   [mq]: Makefile
|
o  0:390c5f52f8e3 beginner
First program

As the fictional character Ford Prefect would say, “Don’t Panic!”.
That is what we were expecting. We’re safe now to pop the patches
off, stay with it: 

$ hg qpop -a
patch queue now empty
$ hg glog --template "{rev}:{node|short} {author|user}\n{desc|firstline}\n\n"
@  6:2048a9e0068b beginner
|  Added README for support.
|
| o  5:db6e2d5b6aa2 Stacey
| |  hg patches saved state
| |
| o  4:cdb5958dcd87 Stacey
| |  imported patch bye
| |
| o  3:d20726d3cc4d Stacey
| |  Fix world
| |
| o  2:63144e5ebe68 Stacey
| |  remove implicit function declaration warnings
| |
| o  1:be53b09aed16 Stacey
|/   [mq]: Makefile
|
o  0:390c5f52f8e3 beginner
First program
$ hg qpop -a -n incoming
using patch queue: /export/home/sm26363/Mercurial/Mercurial/qex1/.hg/incoming
saving bundle to /export/home/sm26363/Mercurial/Mercurial/qex1/.hg/strip-backup/be53b09aed16-temp
adding branch
adding changesets
adding manifests
adding file changes
added 1 changesets with 1 changes to 1 files
patch queue now empty
$ hg qseries -v
0 U Makefile
1 U fix1
2 U message
3 U bye
$ hg qlog
@  changeset:   1:2048a9e0068b
|  tag:         tip
|  user:        beginner
|  date:        Tue Oct 27 21:17:05 2009 +0000
|  summary:     Added README for support.
|
o  changeset:   0:390c5f52f8e3
user:        beginner
date:        Tue Oct 27 21:17:02 2009 +0000
summary:     First program

The above shows the incoming changeset and our patches un-applied.
The patches may now be re-applied: 

$ hg qpush -a
applying Makefile
applying fix1
applying message
applying bye
now at: bye
$ hg qlog
@  changeset:   5:a19f79097ba6
|  tag:         qtip
|  tag:         tip
|  tag:         bye
|  user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
|  date:        Tue Oct 27 21:17:07 2009 +0000
|  summary:     imported patch bye
|
o  changeset:   4:74fc52a79383
|  tag:         message
|  user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
|  date:        Tue Oct 27 21:17:07 2009 +0000
|  summary:     Fix world
|
o  changeset:   3:63ca0f087340
|  tag:         fix1
|  user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
|  date:        Tue Oct 27 21:17:07 2009 +0000
|  summary:     remove implicit function declaration warnings
|
o  changeset:   2:c601487f5539
|  tag:         Makefile
|  tag:         qbase
|  user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
|  date:        Tue Oct 27 21:17:07 2009 +0000
|  summary:     imported patch Makefile
|
o  changeset:   1:2048a9e0068b
|  tag:         qparent
|  user:        beginner
|  date:        Tue Oct 27 21:17:05 2009 +0000
|  summary:     Added README for support.
|
o  changeset:   0:390c5f52f8e3
user:        beginner
date:        Tue Oct 27 21:17:02 2009 +0000
summary:     First program

Preparing to push

The final step is to remove the queues, as with those intact we’re
unable to push. To accomplish this use qfinish But before we
clear the queues take a backup of them using qcommit and clone
as its likely we’ll have to pull again as we raise to push.

Note: Simply doing a qcommit is not sufficient because qfinish tidies
up the patches, a peek into the working files shows this best – I’ve
avoided showing these details previously: 

$ hg qcommit -m "pre-finish"
$ ls .hg/patches
bye       fix1      Makefile  message   series    status
$ hg qfinish -a
patch Makefile finalized without changeset message
patch bye finalized without changeset message
$ ls .hg/patches
series  status
$

By cloning the patches before qfinish we can then recover them after
stripping off our patched changesets, Lets look at another example. 

$ hg qcommit -m 'Pre-finish'
$ hg clone .hg/patches .hg/patches-backup
updating working directory
6 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg qpush -a
all patches are currently applied
$ hg qfinish -a
patch Makefile finalized without changeset message
patch bye finalized without changeset message
$

Now lets assume there is something to pull, we’ll have to remove our
outgoing changes using strip: 

$ hg outgoing
comparing with /export/home/sm26363/Mercurial/Mercurial/qexample
searching for changes
changeset:   2:c601487f5539
user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
date:        Tue Oct 27 21:17:07 2009 +0000
summary:     imported patch Makefile
changeset:   3:63ca0f087340
user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
date:        Tue Oct 27 21:17:07 2009 +0000
summary:     remove implicit function declaration warnings
changeset:   4:74fc52a79383
user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
date:        Tue Oct 27 21:17:07 2009 +0000
summary:     Fix world
changeset:   5:a19f79097ba6
tag:         tip
user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
date:        Tue Oct 27 21:17:07 2009 +0000
summary:     imported patch bye
$ hg strip 2
2 files updated, 0 files merged, 1 files removed, 0 files unresolved
saving bundle to /export/home/sm26363/Mercurial/Mercurial/qex1/.hg/strip-backup/59b69da7d392-backup
$ hg log
changeset:   1:2048a9e0068b
tag:         tip
user:        beginner
date:        Tue Oct 27 21:17:05 2009 +0000
summary:     Added README for support.
changeset:   0:390c5f52f8e3
user:        beginner
date:        Tue Oct 27 21:17:02 2009 +0000
summary:     First program
$

Recover the backups: 

$ rm -rf .hg/patches
$ hg clone .hg/patches-backup .hg/patches
updating working directory
6 files updated, 0 files merged, 0 files removed, 0 files unresolved
$ hg qseries -v
0 U Makefile
1 U fix1
2 U message
3 U bye
$

So there we have it, we can then check the incoming, if it touches
anything we’ve edited then we can go through the merge operation
above and or pull, update re-apply patches, finish and patch.

Changing changeset message

As seen above qfinish nicely told us that some changesets were
missing messages. These can be applied using qrefresh -m: 

$ hg qseries -v
0 U Makefile
1 U fix1
2 U message
3 U bye
$ hg qpush Makefile
applying Makefile
now at: Makefile
$ hg qrefresh -m "Makefile changes"
$ hg qpush bye
applying fix1
applying message
applying bye
now at: bye
$ hg qrefresh -m "Bye: says goodbye"
$ hg glog
@  changeset:   5:202431aaf294
|  tag:         qtip
|  tag:         tip
|  tag:         bye
|  user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
|  date:        Tue Oct 27 22:45:47 2009 +0000
|  summary:     Bye: says goodbye
|
o  changeset:   4:7494b600f084
|  tag:         message
|  user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
|  date:        Tue Oct 27 22:45:21 2009 +0000
|  summary:     Fix world
|
o  changeset:   3:69c047974d13
|  tag:         fix1
|  user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
|  date:        Tue Oct 27 22:45:21 2009 +0000
|  summary:     remove implicit function declaration warnings
|
o  changeset:   2:439652f08146
|  tag:         Makefile
|  tag:         qbase
|  user:        Stacey Marshall ‹Stacey.Marshall@Sun.COM›
|  date:        Tue Oct 27 22:45:15 2009 +0000
|  summary:     Makefile changes
|
o  changeset:   1:2048a9e0068b
|  tag:         qparent
|  user:        beginner
|  date:        Tue Oct 27 21:17:05 2009 +0000
|  summary:     Added README for support.
|
o  changeset:   0:390c5f52f8e3
user:        beginner
date:        Tue Oct 27 21:17:02 2009 +0000
summary:     First program

Note: Obviously for OpenSolaris repository the messages must follow
the bug-id synopsis format.

References

  1. Mq Tutorial
  2. Merging Patches
  3. Multiple changeset pushes to ON – utilising ZFS snapshots for rollback

Last but not least, hg help mq.

Stace

Posted in Tools | 2 Comments »