4248
Comment:

← Revision 93 as of 20140720 19:38:35 ⇥
23232

Deletions are marked like this.  Additions are marked like this. 
Line 1:  Line 1: 
= Use and Contribute to the sagecombinat tree : step by step = Here are the basics steps in order to use and contribute some of your code to sagecombinat. Note that a patch has a unique owner so you should only modify a patch that is already yours. +~1. Install sagecombinat~+ {{{ sage combinat install }}} 1. Familiarize with moving around the patches The sagecombinat is a set of ordered patches. The behavior of moving around those is done like a stack: {{{ hg qpop hg qpush hg qpop a hg qpush a }}} To know which is actually on top or which patches are currently applied or unapplied: {{{ hg qtop hg qapplied hg qunapplied hg qseries }}} Note that after moving around the stack of patches, it is a good idea to rebuild sage before using it : {{{ sage b }}} To display the content of the current top patch, use {{{ hg qdiff # or hg qstatus }}} whether you want a complete diff or simply the name of the modified files. 1. Create a patch Changes are always saved to the actual top patch currently applied. So before doing any changes, you must determine the patch where the changes should be saved. If it is your first contribution, start by creating a patch: {{{ hg qnew my_improvement_AB.patch }}} This creates a new patch right after the current top patch. Usually, a new patch is created on the top of the stack of patches and you can make sure of this by doing: {{{ hg qpush a hg qnew my_improvement_AB.patch }}} It is suggested to add your initials (here AB) in the name so that everybody knows who owns what. TODO : Add more details about name of patch. === 2. Do your modifications === Before making any modifications, make sure the current top patch is yours and is the one you want to add your modifications. {{{ hg qtop }}} Do your modifications to one or many existing files. At any time, you can review your modifications done since the last qrefresh (explained below) or since the creation of the patch by doing: {{{ hg diff }}} or {{{ hg status }}} to list simply the modified files. Note that you can't use qpop and qpush commands once you started modifications. If you added a file, you must specify it by the command: {{{ hg add <filename> }}} === 3. QRefresh the patch === Currently, the modifications are still not part of the patch as seen by the command {{{ hg qdiff hg qstatus }}} that does not display them. Use {{{ hg qrefresh }}} to put the actual modifications in the current top patch. You can see that it worked when typing the command {{{ hg qdiff hg qstatus }}} that should now include your modifications and by the command {{{ hg diff hg status }}} that should not display them anymore. === 4. Do more modifications === You can now use qpop and qpush again and modify other patches you already created. Then follow again 13. === 5. Commit your changes to the local mercurial database === After having done modifications to one or to many patches, you migth want to commit them to the local mercurial database: {{{ hg qcommit }}} It includes all the changes to the patches done since the last commit. An editor will appear for you to provide a description of all the changes you made. === 6. Merge your changes with other sagecombinat developpers === There is a possibility that somebody else pushed changes to the server since the last time you updated your sagecombinat tree. The local mercurial database will be used to merge those modifications. First pull any recent changes on the server by doing : {{{ cd .hg/patches hg pull u }}} and {{{ hg merge }}} if needed. TODO : write about conflicts... === 7. Push your changes to the server === Here is where you must be prudent. Before committing any changes to the server, make sure that sage br works fine: {{{ hg qpop a hg qpush a sage br }}} and that it passes the tests: {{{ sage t <filenames> }}} Then push your server after making sure again that there is no new changes on the server: {{{ cd .hg/patches hg pull u hg push }}} 
#pragma sectionnumbers 2 <#FF0000> '''WARNING''' (July 2014)  <#00FF00>This page is obsolete, as Sage is now using Git instead of Mercurial as a version control system. = The SageCombinat patch server: step by step instructions = This page is meant as a stepbystep tutorial on using the SageCombinat patch server, from basic installation to contributing new patches: <<TableOfContents(3)>> See also: * [[combinat/MercurialTechnical background on the SageCombinat patch server (messy)]] * [[combinat/CodeSharingWorkflowThe rationale behind our workflow]] == Installation and basic usage == === Prerequisites === The instructions below assume that: * You have a recent version of Sage already installed (say in {{{/opt/sage}}}) that can be started by typing {{{sage}}} at the command line. It is recommended to use the most recent stable version, and in particular >= 4.8 as of 2012, April 10th, and even 5.0.beta13 for SageCombinat development) * You have write access to the {{{Sage}}} installation tree * You have internet access to: http://combinat.sagemath.org/patches/ * You have the standard developer tools (compiler, ...) installed. See Sage's [[http://www.sagemath.org/mirror/src/README.txtREADME]] ==== Specific issue on OS X ==== BEFORE running {{{sage combinat install}}}, please open the file {{{$SAGEROOT/local/bin/sagecheck64}}} in the editor of your choice, and comment out ALL lines starting with "echo" by adding "#" in the beginning of the line. {{{#!wiki comment/dotted * Have the [[http://developer.apple.com/technologies/tools/developer tools (XCode)]] installed * Have the newest version of gcc installed (update the unix development tools). * gcc must be callable from within the sage folder $SAGEROOT. This can be done by having the path containing gcc (e.g. $HOME/Developer/usr/bin) added to the variable $PATH. * Doublecheck that you downloaded the appropriate version of Sage (32bit or 64bit). }}} === Downloading and installing the Sagecombinat patches === The patch management is based on the version control system Mercurial, with a support script {{{sage combinat}}} for ease of use. Online help on this script is accessible through: {{{ sage combinat help }}} To install the SageCombinat patches, simply run: {{{ sage combinat install }}} === Updating the SageCombinat patches === To update the SageCombinat patches to the latest version, you can run: {{{ sage combinat update }}} It is possible to simultaneously upgrade Sage to the latest version: {{{ sage combinat upgrade }}} However, we do recommend installing instead a fresh new version of Sage, and then reinstalling the patches on top of it with {{{sage combinat install}}}. === Uninstalling the Sagecombinat patches === {{{ sage b main }}} Now, you may destroy {{{.../sage/devel/sagecombinat/}}} (after checking that you do not have local changes!) == Looking and selecting the patches == === The Sagecombinat stack of patches === Sagecombinat is a collection of experimental patches (i.e. extensions) on top of Sage. Each patch describes a relatively atomic modification which may span several files; it may fix a bug, implement a new feature, improve some documentation. Then main information contained in a patch is a {{{diff}}} file which describes the difference between two files. Here is an example of a {{{diff}}} file. {{{ diff r 01fabd9b3951 sage/combinat/subword.py  a/sage/combinat/subword.py +++ b/sage/combinat/subword.py @@ 34,7 +34,8 @@ def Subwords(w, k=None): """  Returns the combinatorial class of subwords of w. + Returns the combinatorial class of subwords of w. The word w can be given + by either a string or a list. If k is specified, then it returns the combinatorial class of subwords of w of length k. }}} It reads as follows: the first four lines tells that this patch modifies the file {{{sage/combinat/subword.py}}}. The line starting with {{{}}} is replaced by the two lines starting with {{{+}}}. To be able to apply the modifications safely at the right place, some context information is also stored (ie: some more unmodified lines and the position of the modification into the file). The patches are organized as a (totally ordered) stack, each being applied on top of the previous one. The most stable are at the bottom, while the most experimental ones are on top. Let's look at a typical stack of patches: {{{ sage hg qseries }}} It will display something like: {{{ sage3.1.3.patch sage3.2.patch ... lazy_attributes4371nt.patch copy_on_write_fh.patch discrete_functionnt.patch crystals_alcove_path_model_bj.patch words_new_fcts_sl.patch }}} sage3.1.3 is at the bottom of the stack, and is applied first, while words_new_fcts_sl.patch is applied last. The patch sage3.2.patch contains all the sagecombinat patches that have readily been integrated in sage 3.2 (see "using sagecombinat with older versions of sage"). The patch lazy_attributes4371nt.patch is about lazy attributes, related to ticket [[http://trac.sagemath.org/sage_trac/ticket/4371#4371]] on Sage trac, and is owned by NicolasThiéry (nt). In practice, the stack of patches is managed as a Mercurial queue. The command {{{sage hg}}} calls the version of Mercurial that comes with {{{Sage}}}. You may instead simply use {{{hg}}} if Mercurial is installed natively on your system. === Top, applied and unapplied patches === It is possible to move up and down the stack, applying only the patches up to a given one. {{{ sage hg qpush # Apply the first patch in the series which is not currently applied sage hg qpop # Unapply the most recently applied patch sage hg qpush a # Apply all the patches sage hg qpop a # Unapply all the patches }}} If you get confused, the following can tell you which patches are applied or not: {{{ sage hg qtop # Top applied patch sage hg qapplied # Currently applied patches sage hg qunapplied # Currently unapplied patches }}} Note that after moving around the stack of patches, it is necessary to rebuild sage before using it : {{{ sage b sage }}} === Looking inside patches === The description of a patch can be retrieved with: {{{ sage hg qheader # top patch sage ht qheader bla.patch }}} The content of the current top patch can be retrieved with: {{{ sage hg qdiff }}} while the files modified in the top patch can be seen with: {{{ sage hg qstatus }}} Warning: {{{ sage hg qstatus }}} will return irrelevant information if there is no patch currently on top of the stack. === Using the sagecombinat patches with an older version of sage === For the convenience of the user, it is usually possible to use the sagecombinat patches with older versions of sage. The intent is only to temporarily support one or two older versions of sage (that is about one month old). Typical use case: a developer urgently needs the latest version of a patch for a software demonstration at a conference, but can't instantly upgrade because of a slow internet connection. There is no guarantee whatsoever; on occasion we do not support this when this causes technical difficulties. This is the purpose of the patches like {{{sage4.7.patch}}}. This particular patch contains all the sagecombinat patches that have been integrated into Sage 4.7, and is only applied if Sage's version is strictly less than 4.7. This is achieved through guards (see the next section), and is taken care of automatically by {{{sage combinat}}}, and in particular by: {{{ sage combinat qselect }}} === Selecting guarded patches === Some patches may be guarded since they are experimental, not yet finished, or should be only applied for certain versions of sage. Guarded patches are not applied unless explicitly chosen. For example if one would like to apply patches labeled [+experimental] one can use the following steps: {{{ sage hg qselect experimental sage combinat qselect }}} Then reapply the appropriate patches, typically with: {{{ sage hg qpop a sage hg qpush a }}} To disregard all guarded patches one uses instead: {{{ sage hg qselect n sage combinat qselect }}} The following conventions are used regarding positive and negative version guards: * A patch marked +4_7 is not applied if sage's version >= 4_7 Use case: patches merged in 4.7 or that need rebase for 4.7 * A patch marked 4_7 is not applied if sage's version < 4_7 Use case: patches that don't apply anymore on old versions * A patch should not be marked with both +4_7 and 4_7. Starting from Sage 5.0, beta versions of Sage are supported too, using a version guard such as 5.0.beta8 for a patch merged in 5.0.beta8. == Creating and contributing patches == === Mercurial configuration === Write access to the Mercurial server requires authentication using your Sage's trac account username and password; if you do not yet have a Sage's trac account, create one ([[http://trac.sagemath.org/sage_trac/instructions]]). Some further configuration is also required. Open (or create, if it does not exist yet) your Mercurial configuration file ({{{~/.hgrc}}} in your home directory), insert the following, and *edit the username, trac username, password, the shortusername* (usually the 2 initials of your firstname/lastname): {{{ [ui] username = Simon Cussonnet <Simon.Cussonnet at lyceetechnique.thorel.eure.fr> [auth] combinat_patches.prefix = http://combinat.sagemath.org/patches/ combinat_patches.username = <your username on Sage's trac server> combinat_patches.password = <your password on Sage's trac server> [extensions] hgext.mq = hgext.highlight= hgext.record= extdiff = color= [alias] qstatus = status rev 2:. # set the shortusername variable to the letters that define your signature on the patches (e.g. 'sc' for Simon Cussonet) [hooks] preqrefresh = (shortusername='sc' ; if [[ "$(sage hg qtop  grep c "\$shortusername.patch$")" != "1" ]] ; then echo e "\\033[01;05;31m* * * * * * * THIS IS NOT YOUR PATCH * * * * * * *\\033[0;0m" ; fi ; echo "Are you sure you want to refresh the following changes:"; sage hg status; echo n "into the patch: "; sage hg qtop; read p "(y/n)" answer; test "$answer" = "y") preimport = (if [ d $(hg root)/.hg/patches ]; then echo "I'm pretty sure that you don't want to import a changeset !"; echo "Do you mean 'hg qimport <patch>' ?"; exit 1; fi) [extdiff] cmd.interdiff = hginterdiff [diff] nodates=1 showfunc=1 git=1 }}} === Patch naming convention === Each patch should be of the form: {{{ <trac ref if any><theme>_<feature/fix>_<rev if any><owner's shortusername or final or closed>.patch }}} Examples: {{{ trac_9557fundamental_domainsvd.patch sturmian_words_classesabm.patch trac_5991_dynamic_classnt.patch trac_7236_partitions_tableaux_cells_cleanupfh.patch trac_7251_integer_parentnt.patch trac_7251_integer_parentreviewfs.patch }}} A series of patches like the free_modules one above is intended to be progressively folded together into a single patch trac_7251_integer_parentnt.patch before submission to sage (see patch folding below). === Patch description === Each patch starts with a description header which is preserved upon {{{hg export}}}. This is the first thing that will be seen by someone opening the patch. Moreover, the first line of this description will end up into the history of Sage (see the output of {{{sage hg log}}}). So it should be as meaningful as possible, i.e. contain the ticket number associated to the patch and a small description. Typically, this description will match with the summary + description of the (upcoming) trac ticket: {{{ #5431: Free modules: implements blah blah This patch implements ... }}} This description can be edited at any point using {{{hg qrefresh e}}}, so feel free to update it on a regular basis to include progress information (what has readily been implemented, what remains to be done, etc: {{{ #1234: Permutations: implementation of bruhat order Done: * Added methods succ_bruhat, pred_bruhat, ... Todo: * Add method bruhat_order, ... }}} This is usually at this point that one has the best view of this description. === Creating a new patch (qnew) === The first step is to create a new patch. It is easier to create it before doing any modifications to the sage sources. '''Never modify a patch that you do not own.''' {{{ sage hg qnew my_improvement_ab.patch m "#N: description of the patch" }}} where ab are your initials and N is the associated sage trac ticket number if it exists. For a larger description you may want to use the e option to edit the description with your favorite editor. The new patch is created on top of the most recently applied patch. You may use {{{qpush}}} and {{{qpop}}} first to choose where your patch is created. If you accidentally made changes before creating a new patch, the command {{{ sage hg qnew f my_improvement_ab.patch m "#N: description of the patch" }}} will create the new patch including the current 'orphaned' changes into it. === Editing the Sage sources === It is recommended to double check that the current top patch is yours and is the one you want to add your modifications to (see {{{qtop}}} and {{{qpop/qpush}}}). '''using {{{qpop}}} and {{{qpush}}} become tricky to use once you started modifications.''' Now you can edit the Sage sources files to your taste in {{{$SAGE_ROOT/devel/sagecombinat/sage/}}}. At any time, you can review your modifications since the creation of the patch (or since the last {{{qrefresh}}}, see below) by doing: {{{ sage hg diff #complete modifications since last qrefresh sage hg status #list the modified files since last qrefresh }}} If you added a new source file, you must declare it using the command: {{{ sage hg add <filename> }}} === Refreshing your patch (qrefresh) === Currently, the modifications are still not part of the patch as may be seen with the commands {{{ sage hg diff # Completely lists the modifications sage hg status # Lists only the affected files }}} Note that {{{sage hg status}}} gives the modifications which are not part of the current patch, while {{{sage hg qstatus}}} gives the modifications which are part of the current patch (and similarly for {{{diff}}} and {{{qdiff}}}). Use {{{ sage hg qrefresh }}} to put the actual modifications in the current top patch. Tip: on a regular basis, use also: {{{ sage hg qrefresh e }}} OR {{{ sage hg qrefresh m "#N: description of the patch" }}} to update the description of the patch, as this usually is the point in time where one has the best view of what it should be. It can also be the opportunity to add the sage trac ticket number to the description if it is created. You can check that the modifications have been included in the patch with: {{{ sage hg qdiff sage hg qstatus }}} Also, they should not appear anymore in: {{{ sage hg diff sage hg status }}} After {{{qrefresh}}}, you can use {{{qpop}}} and {{{qpush}}} again and modify the same patch or some other patch you already created. === Removing a patch === In case you want to discard your patch, you may use: {{{ sage hg qremove my_improvement_AB.patch }}} You may use {{{ sage hg qseries }}} to confirm that the patch is removed. === Pushing patches to the sagecombinat server === Go to the sagecombinat directory, and make sure that there is no local changes: {{{ cd $SAGE_ROOT/devel/sagecombinat sage hg status }}} Otherwise discard them ({{{sage hg revert}}}) or save them in your favorite patch ({{{sage hg qrefresh}}}). Pull the latest version of the patches from the server, and make sure that everything is working fine after applying all patches: {{{ sage combinat update f sage hg qpush a sage br # Make sure this works sage t filenames # If you believe your tests should pass, check them. }}} The {{{f}}} is to tell {{{sage combinat}}} to run even if your patch directory is modified, which it probably is. When pulling from the server, your changes will be merged with those from the server. You may get conflicts in the series file, or patches that do not apply anymore. Fix the patches that you own, and get in touch with the owners of the other broken patches. When everything is ready, double check that you are up to date: {{{ sage combinat update f }}} and then: {{{ cd $SAGE_ROOT/devel/sagecombinat/.hg/patches sage hg commit sage hg push }}} In case you have been unlucky, and there has been a change on the patch server between your last pull and your commit, push will fail with an error like "abort: push creates new remote heads! did you forget to merge?". You then have to merge your local modifications with the remote ones: {{{ cd .hg/patches sage hg pull u sage hg merge }}} If the merge goes through automatically, just commit with "Merge" as message, and push. Otherwise, read the section Merging conflicts and feel free to ask for help! === In the case of Merging conflicts === In the case of merging conflicts, you should obtain such a message : {{{ TOBEDONE }}} First, use the following to list the files needing a manual merge. If you edited your own patches and nobody edited your patches, only the series file should be listed. If other files are listed, you should definitively ask for help. {{{ cd .hg/patches hg resolve l #list state of files needing merge (U = unresolve, R = resolved) }}} Open and edit the series file and merge the modifications yourself. Then, when you are done, mark the series file as resolve by doing {{{ hg resolve m series }}} Make sure the series file is now marked as resolved using: {{{ hg resolve l #list state of files needing merge (U = unresolve, R = resolved) }}} Then, try to merge again {{{ hg merge }}} and commit if the merge worked: {{{ hg commit }}} You may now try to push again to the sagecombinat server as explain above. === Handling rejection === Sometimes a patch fails to apply and you get, after a {{{ sage hg qpush }}} a message like : {{{ applying my_modification.patch patching file file_a Hunk #1 FAILED at 0 1 out of 17 hunks FAILED  saving rejects to file file_a.rej patch failed, unable to continue (try v) patch failed, rejects left in working dir Errors during apply, please fix and refresh my_modification.patch }}} After such an error, Mercurial has made all modification its can to the file file_a and ''rejects'' the hunk its could not understand. All modifications inside the patch and not understood by Mercurial are in the file file_a.rej created in the working directory (e.g. at the same place that file_a is). The rejected file is just a part of the initial patch. At this point you have several solutions, the first is more simple : * make by hand the modification in file_a.rej to file_a and refresh the patch my_modification.patch * try to reorganize directly the patch my_modification.patch ('''DANGEROUS''') * use the Chris Mason's [[http://oss.oracle.com/~mason/mpatch/mpatch]] to try to reorganize the patch (automatization of the preceding solution). In any case be wise with any modification ! === Exporting Patches for use with trac === When a patch is ready to be applied to {{{Sage}}}, you should first verify that it will apply cleanly. If you want to be able to undo anything you do in this step, you should run {{{sage hg qcommit }}} before beginning. The first step is, from the {{{PATCHES}}} directory, manually edit the {{{series }}} to move your patch in the queue, so that your patch (or patches) are applied only to patches labelled *_closed. Then (from the source directories) use {{{qpush}}} and {{{qpop}}} to bring the first of your patches to the top of the stack (ie, only one of your patches should be applied). Suppose your patches were called 1.patch and 2.patch, and you currently have 1.patch at the top of the stack, and 2.patch as the next patch to be applied. The command {{{sage hg qfold 2.patch}}} will incorporate the changes from 2.patch into 1.patch, and then delete 2.patch. You should repeat this process until all patches working on one particular issue have been incorporated. Double check and update the description of the patch, using {{{hg qrefresh e}}} and make sure to add the sage trac ticket number to the first line of the description. You may also rename the patch (using {{{sage hg qrename}}}) by appending {{{trac_####}}} as a prefix. Then you should make sure that sage builds, runs, and passes tests, with just this patch applied on top of the patches that have already been closed. Don't forget to also commit and push your changes to the server. The next step is to export your patch for submission. Running {{{sage hg export}}} includes extra information in the patch such as the author of the patch. {{{ cd $SAGE_ROOT/devel/sagecombinat sage hg export new_feature.patch > /path/to/new_feature.patch }}} Upload the file new_feature_for_trac.patch to the appropriate ticket on the Sage Trac server, creating the ticket if necessary. Generic procedures for using the trac system for Sage are described here: http://wiki.sagemath.org/TracGuidelines . Here are some further tips for SageCombinat tickets: * Keep the same patch name in the SageCombinat queue and on trac. * Assign it to milestone sagecombinat initially. Once it has a positive review it should be moved to the current merge milestone that is active. * Add "sagecombinat" to the CC field. That way ticket status changes show up at http://groups.google.com/group/sagecombinatcommits == Rebasing the patch queue on a new version of sage == This section is for maintainers of the patch queue. Idealy, switching to a new version of Sage should require no work. But sometimes some patches from the Sage community at large that were integrated in the new version of Sage may conflict with the SageCombinat patches. Here is how to rebase on them. In this example, we rebase from sage 3.4 to sage 3.4.1. This requires having access to vanilla sage/devel/sagemain directories for 3.4 and 3.4.1 which we assume to be respectively in {{{/opt/sage/devel/sagemain3.4}}} and {{{/opt/sage/devel/sagemain}}} Start from a fresh sagemain from the previous sage version: {{{ cd /tmp hg clone /opt/sage/devel/sagemain3.4 sagemain }}} Install the sagecombinat patches, and apply them all {{{ cd sagemain/.hg hg clone http://combinat.sagemath.org/patches patches cd .. hg qselect 3_4_1 hg qpush a }}} Then we save the queue state, and launch a 3way merge {{{ hg qsave e c hg pull /opt/sage/devel/sagemain hg update C tip hg qselect n hg qpush m a }}} 
WARNING (July 2014) 
This page is obsolete, as Sage is now using Git instead of Mercurial as a version control system. 
The SageCombinat patch server: step by step instructions
This page is meant as a stepbystep tutorial on using the SageCombinat patch server, from basic installation to contributing new patches:
Contents
 Installation and basic usage
 Looking and selecting the patches

Creating and contributing patches
 Mercurial configuration
 Patch naming convention
 Patch description
 Creating a new patch (qnew)
 Editing the Sage sources
 Refreshing your patch (qrefresh)
 Removing a patch
 Pushing patches to the sagecombinat server
 In the case of Merging conflicts
 Handling rejection
 Exporting Patches for use with trac
 Rebasing the patch queue on a new version of sage
See also:
1. Installation and basic usage
1.1. Prerequisites
The instructions below assume that:
 You have a recent version of Sage already installed (say in
/opt/sage) that can be started by typing sage at the command line. It is recommended to use the most recent stable version, and in particular >= 4.8 as of 2012, April 10th, and even 5.0.beta13 for SageCombinat development)
You have write access to the Sage installation tree
You have internet access to: http://combinat.sagemath.org/patches/
 You have the standard developer tools (compiler, ...) installed.
See Sage's README
1.1.1. Specific issue on OS X
BEFORE running sage combinat install, please open the file $SAGEROOT/local/bin/sagecheck64 in the editor of your choice, and comment out ALL lines starting with "echo" by adding "#" in the beginning of the line.
1.2. Downloading and installing the Sagecombinat patches
The patch management is based on the version control system Mercurial, with a support script sage combinat for ease of use. Online help on this script is accessible through:
sage combinat help
To install the SageCombinat patches, simply run:
sage combinat install
1.3. Updating the SageCombinat patches
To update the SageCombinat patches to the latest version, you can run:
sage combinat update
It is possible to simultaneously upgrade Sage to the latest version:
sage combinat upgrade
However, we do recommend installing instead a fresh new version of Sage, and then reinstalling the patches on top of it with sage combinat install.
1.4. Uninstalling the Sagecombinat patches
sage b main
Now, you may destroy .../sage/devel/sagecombinat/ (after checking that you do not have local changes!)
2. Looking and selecting the patches
2.1. The Sagecombinat stack of patches
Sagecombinat is a collection of experimental patches (i.e. extensions) on top of Sage. Each patch describes a relatively atomic modification which may span several files; it may fix a bug, implement a new feature, improve some documentation. Then main information contained in a patch is a diff file which describes the difference between two files. Here is an example of a diff file.
diff r 01fabd9b3951 sage/combinat/subword.py  a/sage/combinat/subword.py +++ b/sage/combinat/subword.py @@ 34,7 +34,8 @@ def Subwords(w, k=None): """  Returns the combinatorial class of subwords of w. + Returns the combinatorial class of subwords of w. The word w can be given + by either a string or a list. If k is specified, then it returns the combinatorial class of subwords of w of length k.
It reads as follows: the first four lines tells that this patch modifies the file sage/combinat/subword.py. The line starting with  is replaced by the two lines starting with +. To be able to apply the modifications safely at the right place, some context information is also stored (ie: some more unmodified lines and the position of the modification into the file).
The patches are organized as a (totally ordered) stack, each being applied on top of the previous one. The most stable are at the bottom, while the most experimental ones are on top. Let's look at a typical stack of patches:
sage hg qseries
It will display something like:
sage3.1.3.patch sage3.2.patch ... lazy_attributes4371nt.patch copy_on_write_fh.patch discrete_functionnt.patch crystals_alcove_path_model_bj.patch words_new_fcts_sl.patch
sage3.1.3 is at the bottom of the stack, and is applied first, while words_new_fcts_sl.patch is applied last. The patch sage3.2.patch contains all the sagecombinat patches that have readily been integrated in sage 3.2 (see "using sagecombinat with older versions of sage"). The patch lazy_attributes4371nt.patch is about lazy attributes, related to ticket #4371 on Sage trac, and is owned by NicolasThiéry (nt).
In practice, the stack of patches is managed as a Mercurial queue. The command sage hg calls the version of Mercurial that comes with Sage. You may instead simply use hg if Mercurial is installed natively on your system.
2.2. Top, applied and unapplied patches
It is possible to move up and down the stack, applying only the patches up to a given one.
sage hg qpush # Apply the first patch in the series which is not currently applied sage hg qpop # Unapply the most recently applied patch sage hg qpush a # Apply all the patches sage hg qpop a # Unapply all the patches
If you get confused, the following can tell you which patches are applied or not:
sage hg qtop # Top applied patch sage hg qapplied # Currently applied patches sage hg qunapplied # Currently unapplied patches
Note that after moving around the stack of patches, it is necessary to rebuild sage before using it :
sage b sage
2.3. Looking inside patches
The description of a patch can be retrieved with:
sage hg qheader # top patch sage ht qheader bla.patch
The content of the current top patch can be retrieved with:
sage hg qdiff
while the files modified in the top patch can be seen with:
sage hg qstatus
Warning: sage hg qstatus will return irrelevant information if there is no patch currently on top of the stack.
2.4. Using the sagecombinat patches with an older version of sage
For the convenience of the user, it is usually possible to use the sagecombinat patches with older versions of sage. The intent is only to temporarily support one or two older versions of sage (that is about one month old). Typical use case: a developer urgently needs the latest version of a patch for a software demonstration at a conference, but can't instantly upgrade because of a slow internet connection. There is no guarantee whatsoever; on occasion we do not support this when this causes technical difficulties.
This is the purpose of the patches like sage4.7.patch. This particular patch contains all the sagecombinat patches that have been integrated into Sage 4.7, and is only applied if Sage's version is strictly less than 4.7. This is achieved through guards (see the next section), and is taken care of automatically by sage combinat, and in particular by:
sage combinat qselect
2.5. Selecting guarded patches
Some patches may be guarded since they are experimental, not yet finished, or should be only applied for certain versions of sage. Guarded patches are not applied unless explicitly chosen. For example if one would like to apply patches labeled [+experimental] one can use the following steps:
sage hg qselect experimental sage combinat qselect
Then reapply the appropriate patches, typically with:
sage hg qpop a sage hg qpush a
To disregard all guarded patches one uses instead:
sage hg qselect n sage combinat qselect
The following conventions are used regarding positive and negative version guards:
A patch marked +4_7 is not applied if sage's version >= 4_7
 Use case: patches merged in 4.7 or that need rebase for 4.7
A patch marked 4_7 is not applied if sage's version < 4_7
 Use case: patches that don't apply anymore on old versions
 A patch should not be marked with both +4_7 and 4_7.
Starting from Sage 5.0, beta versions of Sage are supported too, using a version guard such as 5.0.beta8 for a patch merged in 5.0.beta8.
3. Creating and contributing patches
3.1. Mercurial configuration
Write access to the Mercurial server requires authentication using your Sage's trac account username and password; if you do not yet have a Sage's trac account, create one (instructions).
Some further configuration is also required. Open (or create, if it does not exist yet) your Mercurial configuration file (~/.hgrc in your home directory), insert the following, and *edit the username, trac username, password, the shortusername* (usually the 2 initials of your firstname/lastname):
[ui] username = Simon Cussonnet <Simon.Cussonnet at lyceetechnique.thorel.eure.fr> [auth] combinat_patches.prefix = http://combinat.sagemath.org/patches/ combinat_patches.username = <your username on Sage's trac server> combinat_patches.password = <your password on Sage's trac server> [extensions] hgext.mq = hgext.highlight= hgext.record= extdiff = color= [alias] qstatus = status rev 2:. # set the shortusername variable to the letters that define your signature on the patches (e.g. 'sc' for Simon Cussonet) [hooks] preqrefresh = (shortusername='sc' ; if [[ "$(sage hg qtop  grep c "\$shortusername.patch$")" != "1" ]] ; then echo e "\\033[01;05;31m* * * * * * * THIS IS NOT YOUR PATCH * * * * * * *\\033[0;0m" ; fi ; echo "Are you sure you want to refresh the following changes:"; sage hg status; echo n "into the patch: "; sage hg qtop; read p "(y/n)" answer; test "$answer" = "y") preimport = (if [ d $(hg root)/.hg/patches ]; then echo "I'm pretty sure that you don't want to import a changeset !"; echo "Do you mean 'hg qimport <patch>' ?"; exit 1; fi) [extdiff] cmd.interdiff = hginterdiff [diff] nodates=1 showfunc=1 git=1
3.2. Patch naming convention
Each patch should be of the form:
<trac ref if any><theme>_<feature/fix>_<rev if any><owner's shortusername or final or closed>.patch
Examples:
trac_9557fundamental_domainsvd.patch sturmian_words_classesabm.patch trac_5991_dynamic_classnt.patch trac_7236_partitions_tableaux_cells_cleanupfh.patch trac_7251_integer_parentnt.patch trac_7251_integer_parentreviewfs.patch
A series of patches like the free_modules one above is intended to be progressively folded together into a single patch trac_7251_integer_parentnt.patch before submission to sage (see patch folding below).
3.3. Patch description
Each patch starts with a description header which is preserved upon hg export. This is the first thing that will be seen by someone opening the patch. Moreover, the first line of this description will end up into the history of Sage (see the output of sage hg log). So it should be as meaningful as possible, i.e. contain the ticket number associated to the patch and a small description.
Typically, this description will match with the summary + description of the (upcoming) trac ticket:
#5431: Free modules: implements blah blah This patch implements ...
This description can be edited at any point using hg qrefresh e, so feel free to update it on a regular basis to include progress information (what has readily been implemented, what remains to be done, etc:
#1234: Permutations: implementation of bruhat order Done: * Added methods succ_bruhat, pred_bruhat, ... Todo: * Add method bruhat_order, ...
This is usually at this point that one has the best view of this description.
3.4. Creating a new patch (qnew)
The first step is to create a new patch. It is easier to create it before doing any modifications to the sage sources. Never modify a patch that you do not own.
sage hg qnew my_improvement_ab.patch m "#N: description of the patch"
where ab are your initials and N is the associated sage trac ticket number if it exists. For a larger description you may want to use the e option to edit the description with your favorite editor.
The new patch is created on top of the most recently applied patch. You may use qpush and qpop first to choose where your patch is created. If you accidentally made changes before creating a new patch, the command
sage hg qnew f my_improvement_ab.patch m "#N: description of the patch"
will create the new patch including the current 'orphaned' changes into it.
3.5. Editing the Sage sources
It is recommended to double check that the current top patch is yours and is the one you want to add your modifications to (see qtop and qpop/qpush). using qpop and qpush become tricky to use once you started modifications.
Now you can edit the Sage sources files to your taste in $SAGE_ROOT/devel/sagecombinat/sage/. At any time, you can review your modifications since the creation of the patch (or since the last qrefresh, see below) by doing:
sage hg diff #complete modifications since last qrefresh sage hg status #list the modified files since last qrefresh
If you added a new source file, you must declare it using the command:
sage hg add <filename>
3.6. Refreshing your patch (qrefresh)
Currently, the modifications are still not part of the patch as may be seen with the commands
sage hg diff # Completely lists the modifications sage hg status # Lists only the affected files
Note that sage hg status gives the modifications which are not part of the current patch, while sage hg qstatus gives the modifications which are part of the current patch (and similarly for diff and qdiff).
Use
sage hg qrefresh
to put the actual modifications in the current top patch.
Tip: on a regular basis, use also:
sage hg qrefresh e
OR
sage hg qrefresh m "#N: description of the patch"
to update the description of the patch, as this usually is the point in time where one has the best view of what it should be. It can also be the opportunity to add the sage trac ticket number to the description if it is created.
You can check that the modifications have been included in the patch with:
sage hg qdiff sage hg qstatus
Also, they should not appear anymore in:
sage hg diff sage hg status
After qrefresh, you can use qpop and qpush again and modify the same patch or some other patch you already created.
3.7. Removing a patch
In case you want to discard your patch, you may use:
sage hg qremove my_improvement_AB.patch
You may use
sage hg qseries
to confirm that the patch is removed.
3.8. Pushing patches to the sagecombinat server
Go to the sagecombinat directory, and make sure that there is no local changes:
cd $SAGE_ROOT/devel/sagecombinat sage hg status
Otherwise discard them (sage hg revert) or save them in your favorite patch (sage hg qrefresh).
Pull the latest version of the patches from the server, and make sure that everything is working fine after applying all patches:
sage combinat update f sage hg qpush a sage br # Make sure this works sage t filenames # If you believe your tests should pass, check them.
The f is to tell sage combinat to run even if your patch directory is modified, which it probably is.
When pulling from the server, your changes will be merged with those from the server. You may get conflicts in the series file, or patches that do not apply anymore. Fix the patches that you own, and get in touch with the owners of the other broken patches.
When everything is ready, double check that you are up to date:
sage combinat update f
and then:
cd $SAGE_ROOT/devel/sagecombinat/.hg/patches sage hg commit sage hg push
In case you have been unlucky, and there has been a change on the patch server between your last pull and your commit, push will fail with an error like "abort: push creates new remote heads! did you forget to merge?". You then have to merge your local modifications with the remote ones:
cd .hg/patches sage hg pull u sage hg merge
If the merge goes through automatically, just commit with "Merge" as message, and push. Otherwise, read the section Merging conflicts and feel free to ask for help!
3.9. In the case of Merging conflicts
In the case of merging conflicts, you should obtain such a message :
TOBEDONE
First, use the following to list the files needing a manual merge. If you edited your own patches and nobody edited your patches, only the series file should be listed. If other files are listed, you should definitively ask for help.
cd .hg/patches hg resolve l #list state of files needing merge (U = unresolve, R = resolved)
Open and edit the series file and merge the modifications yourself. Then, when you are done, mark the series file as resolve by doing
hg resolve m series
Make sure the series file is now marked as resolved using:
hg resolve l #list state of files needing merge (U = unresolve, R = resolved)
Then, try to merge again
hg merge
and commit if the merge worked:
hg commit
You may now try to push again to the sagecombinat server as explain above.
3.10. Handling rejection
Sometimes a patch fails to apply and you get, after a
sage hg qpush
a message like :
applying my_modification.patch patching file file_a Hunk #1 FAILED at 0 1 out of 17 hunks FAILED  saving rejects to file file_a.rej patch failed, unable to continue (try v) patch failed, rejects left in working dir Errors during apply, please fix and refresh my_modification.patch
After such an error, Mercurial has made all modification its can to the file file_a and rejects the hunk its could not understand. All modifications inside the patch and not understood by Mercurial are in the file file_a.rej created in the working directory (e.g. at the same place that file_a is). The rejected file is just a part of the initial patch.
At this point you have several solutions, the first is more simple :
 make by hand the modification in file_a.rej to file_a and refresh the patch my_modification.patch
try to reorganize directly the patch my_modification.patch (DANGEROUS)
use the Chris Mason's mpatch to try to reorganize the patch (automatization of the preceding solution).
In any case be wise with any modification !
3.11. Exporting Patches for use with trac
When a patch is ready to be applied to Sage, you should first verify that it will apply cleanly. If you want to be able to undo anything you do in this step, you should run sage hg qcommit before beginning.
The first step is, from the PATCHES directory, manually edit the series to move your patch in the queue, so that your patch (or patches) are applied only to patches labelled *_closed. Then (from the source directories) use qpush and qpop to bring the first of your patches to the top of the stack (ie, only one of your patches should be applied). Suppose your patches were called 1.patch and 2.patch, and you currently have 1.patch at the top of the stack, and 2.patch as the next patch to be applied. The command sage hg qfold 2.patch will incorporate the changes from 2.patch into 1.patch, and then delete 2.patch. You should repeat this process until all patches working on one particular issue have been incorporated.
Double check and update the description of the patch, using hg qrefresh e and make sure to add the sage trac ticket number to the first line of the description. You may also rename the patch (using sage hg qrename) by appending trac_#### as a prefix.
Then you should make sure that sage builds, runs, and passes tests, with just this patch applied on top of the patches that have already been closed. Don't forget to also commit and push your changes to the server.
The next step is to export your patch for submission. Running sage hg export includes extra information in the patch such as the author of the patch.
cd $SAGE_ROOT/devel/sagecombinat sage hg export new_feature.patch > /path/to/new_feature.patch
Upload the file new_feature_for_trac.patch to the appropriate ticket on the Sage Trac server, creating the ticket if necessary. Generic procedures for using the trac system for Sage are described here: http://wiki.sagemath.org/TracGuidelines . Here are some further tips for SageCombinat tickets:
 Keep the same patch name in the SageCombinat queue and on trac.
 Assign it to milestone sagecombinat initially. Once it has a positive review it should be moved to the current merge milestone that is active.
Add "sagecombinat" to the CC field. That way ticket status changes show up at http://groups.google.com/group/sagecombinatcommits
4. Rebasing the patch queue on a new version of sage
This section is for maintainers of the patch queue. Idealy, switching to a new version of Sage should require no work. But sometimes some patches from the Sage community at large that were integrated in the new version of Sage may conflict with the SageCombinat patches. Here is how to rebase on them. In this example, we rebase from sage 3.4 to sage 3.4.1. This requires having access to vanilla sage/devel/sagemain directories for 3.4 and 3.4.1 which we assume to be respectively in /opt/sage/devel/sagemain3.4 and /opt/sage/devel/sagemain
Start from a fresh sagemain from the previous sage version:
cd /tmp hg clone /opt/sage/devel/sagemain3.4 sagemain
Install the sagecombinat patches, and apply them all
cd sagemain/.hg hg clone http://combinat.sagemath.org/patches patches cd .. hg qselect 3_4_1 hg qpush a
Then we save the queue state, and launch a 3way merge
hg qsave e c hg pull /opt/sage/devel/sagemain hg update C tip hg qselect n hg qpush m a