Mercurial Queues and the combinat server

The suggested process for people doing significant development to the combinat area of sage is to work in a clone of a clean version of sage (this clone is typically called sage-combinat). The files in this clone are managed with a Mercurial repository using Mercurial queues. The patch files that are created with this process are managed in a secondary Mercurial repository, which is periodically synched with the server at sage.math.washington.edu:2144 . The purpose of this page is to give more detail on how this process works. General concepts are given at the top, and examples of common working situations are given at the bottom. To skip to one of these common examples, click on one of the following links.

Important Sage Concepts

There are two important sage commands that you must be aware of as part of the development process. The first is related to the fact that editing the sage source-code files will not immediately change the behaviour of sage. In order for your changes to have an effect, you must first run

$ sage -b combinat

which will rebuild combinat branch of sage, using the current source code files. Note that the 'combinat' argument is not necessary if the last time you ran sage you used the combinat branch. Variants of this command are  $ sage -br  (build and run) which will build sage and then run it, and  $ sage -ba  (build all) which is necessary if any Cython files have been modified.

The second important sage command to know is

$ sage -t filenames

which will run all the doctests in named files. Note that you should rebuild sage before running  $ sage -t .

Managing the Primary Repository

The primary Mercurial repository manages all files contained in  $SAGEROOT/devel/sage-combinat/ (including all subdirectories) EXCEPT the directory  $SAGEROOT/devel/sage-combinat/.hg/patches . The files in that directory are managed by the secondary repository which is discussed below. The files in the primary repository are all the files that make sage work, so it is important to manage any changes that are made to these files. That way your changes can easily be undone, shared with other people, etc.

Patch and Queue Concepts

The primary repository is managed with a Mercurial queue of patches. A patch is information that tells Mercurial how to go back and forth between an "old version" of a collection of files and a "new version" of those files. Changing the collection from the old version to the new is called applying the patch, and the inverse is called unapplying the patch. The information about how to make this transformation is stored in a text file with a .patch extension. For our project, all patch files live in the directory $SAGEROOT/devel/sage-comibinat/.hg/patches . Since these are just text files, they are easy to manually inspect with less or any text editor. Manually editing these files, however, is strongly not recommended.

Our project consists of many patches, and these patches are managed with a Mercurial queue. A queue can be thought of as a "stack" of patches. Every change that we make to sage should belong to a patch, and every patch should belong to the queue. If, from any subdirectory of  $SAGEROOT/devel/sage-combinat/ (except .hg/patches you type the command  $ hg qseries  you will see a list of patches in the queue. These patches are ordered, with the first in the list being the first patch which is applied to the base version of sage. Mercurial stores the series in the text file $SAGEROOT/devel/sage-comibinat/.hg/patches/series . If for some reason you want to change the order in which patches are applied, it is possible to edit this file manually, though this is not generally recommended. (It may be necessary, however, when updating the secondary repository; see below for more details). It is important to realize that not all of the patches in the series are necessarily currently applied. You can see which of these patches are currently applied with the command $ hg qapplied , or just get the name of the last patch applied with the command $ hg top .

What makes working with Mercurial queues so nice, is that it makes applying and unapplying patches very, very easy. Specifically, the command $ hg qpop  will unapply the patch currently on the top of the stack, and $ hg qpush  will apply the next patch in the series that is not currently applied. The variants $ hg qpop -a and $ hg qpush -a will unapply and apply (respectively) all patches in the series. So, for example, if you type $ hg qpop -a all files will revert to their state as of the base version of sage. You can now rebuild and run sage to see sage as ordinary users see it. When you get bored of this and want to go back to the exciting bleeding-edge development version, it's as simple as $ hg qpush -a and $ sage -br.

Creating a new patch

This is done with the command $ hg qnew patchname.patch. Generally speaking, you should only create a new patch at the top of the stack, with all previous patches applied. In other words, you should always run $ hg qpush -a before running $ hg qnew . Furthermore, it is a good idea to synch your queue with the server (see below) before starting a new patch. After an $ hg qnew , you will see the new patch as the top patch with $ hg qseries , $ hg qapplied , and $ hg qtop .

Editing Patches

The basic idea in editing patches is that you should move the patch you want to edit to the top of the stack (using qpush, qpop, or qnew), edit the files you want to edit, then update the current patch file with the command  $ hg qrefresh . If you only want to update your changes to particular files, you can do this with  $ hg qrefresh -I filenames . (More options to this command can be found with  $ hg help qrefresh .) Once you start editing files, Mercurial will not let you qpush or qpop until you have done a qrefresh. Obviously, you should not edit a patch in the middle of the series if you know there will be conflicts with patches later in the series. However, if you have minor changes that logically belong to a patch in the middle of the stack, it is encouraged to make your edits to that patch, as opposed to creating a new one.

If in your editing process you create or delete files, you must give this information to Mercurial. This is done with the commands $ hg add filename and $ hg remove filename, respectively.

Since it is easy to forget your current state of editing, Mercurial provides commands to show you what you have done. The command $ hg status  will show you what files you have edited since the last time you did $ hg qrefresh . The command $ hg diff  will show you precisely what these changes are.

If you have made edits since your last qrefresh that you've decided you don't want, you can get rid of them with $ hg revert filenames or hg revert --all . This will take the indicated files back to their state at the time of the last qrefresh. Note that the edits since that time will be completely lost.

You may notice that our instructions to this point only provide a means for having a single saved version of any given patch. It is possible to store information about multiple versions of a single patch file; this is done with the secondary repository which is discussed in more detail below.

Quick Summary of Common Queue Commands

$ hg help -v           # List all Mercurial commands (including queue commands)
$ hg help command      # Provides a brief description and available options for the named command
$ hg qseries           # Lists all patches in the queue, starting with the first to be applied
$ hg qapplied          # Lists all patches in the queue which are currently applied
$ hg qtop              # Displays the most recently applied patch
$ hg qpush             # Apply the first patch in the series which is not currently applied
$ hg pop               # Unapply the most recently applied patch
$ hg qnew name.patch   # Create a new patch on top of the current 'qtop'
$ hg add filename      # Add a file to the Mercurial repository
$ hg remove filename   # Remove a file from the Mercurial repository
$ hg status            # List of files modified since last 'qrefresh'
$ hg diff              # List of precise changes since last 'qrefresh'
$ hg qrefresh          # Store recent changes in the current top patch
$ hg revert filenames  # Lose all changes to filenames since last 'qrefresh'
$ hg remove name.patch # Completely remove the named patch from the entire queue. The patch must currently be unapplied

Managing the Secondary Repository

At this point, you should understand how to manage your changes to the sage source files using a collection of patch files managed by a Mercurial queue. However

Mercurial Concepts

Mercurial Commands

Example

xxx

Example of common situations

Create a combinat branch and synch with server

Example

xxx

Pull changes from server and merge with local changes

Example

Push changes to server

Example

Resolving Conflicts

Example

Exporting Patches for use with trac

Example

Upgrading to new version of sage

Example