|
Size: 18865
Comment: removed menas as no longer on skynet
|
← Revision 48 as of 2022-03-31 02:28:10 ⇥
Size: 0
Comment: Outdated
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| ## page was renamed from release = Sage Release Management = The first rule of release management, is that the release manager is in charge. They get to decide on whether or not ''any'' ticket goes into the next Sage release, gets closed, etc. If they feel like not merging a particular patch, spkg, or contribution from somebody, that is their right. If you don't like it, wait for another release manager. (Of course, the BDFL has final say.) <<TableOfContents>> == Overview == These are some notes on various aspects of the Sage release process. It is also worth reading Eric S. Raymond's article [[http://www.tldp.org/HOWTO/Software-Release-Practice-HOWTO/|Software Release Practice HOWTO]]. The following serve as introductory materials to beginning release managers: * [[http://mvngu.wordpress.com/2010/02/09/release-management-merging-patches/ | Release Management: Merging Patches]] --- An overview of merging patches. * [[http://mvngu.wordpress.com/2010/02/09/automatically-generating-a-release-note-template/ | Automatically Generating A Release Note Template]] --- Producing a release note. * See also [[http://groups.google.com/group/sage-release/browse_thread/thread/4c915ae814dd6514 | this thread]] on the sage-release mailing list. == sage -merge/sage-apply-ticket == This is a new script written by Craig Citro and Nick Alexander. It tries to automate much of the process of downloading patches from trac, merging them, and running doctests. Here's a quick rundown of usage: * `sage -merge -c` or `sage -merge --candidates` List all candidates for merging, i.e. all trac tickets with positive review. * `sage -merge <ticket_number> <options>` Download patches from trac for the given ticket number, merge them, run tests, and report the results. * `sage -merge -a <options>` or `sage -merge --all <options>` For each ticket on trac with a positive review, download the patches, apply them, and run doctests. At the end, report which tickets passed, which failed, and which didn't have any files to doctest (as is commonly the case with tickets for new spkg's). === Notes === * This script makes heavy use of the queues extension (mq) to Mercurial. It is recommended that you understand how mq works and know how to use it for managing patches. This is especially relevant if you consider using the merge script for developing Sage or managing a release. For an introductory tutorial on mq, see the [[http://mercurial.selenic.com/wiki/MqTutorial|Mq Tutorial]] from Selenic. Bryan O'Sullivan's book [[http://hgbook.red-bean.com/read/|"Mercurial: The Definitive Guide"]] is a handy reference on using Mercurial for everyday development. Chapters [[http://hgbook.red-bean.com/read/managing-change-with-mercurial-queues.html|12]] and [[http://hgbook.red-bean.com/read/advanced-uses-of-mercurial-queues.html|13]] carefully explain how to use mq to manage patches. [[http://hgbook.red-bean.com/read/mercurial-queues-reference.html|Appendix B]] of the book is a reference on features of mq. * Currently, the merge script requires that the patch queue be '''empty''' to start. After any test, it will pop all changes, regardless of outcome, unless an option (as below) asks it to do otherwise. Hint: {{{hg update && hg qpop -a && hg qrm `hg qser`}}} kills the queue. * If there are multiple patches on a ticket, the merge script will spawn an editor and allow you to choose which patches should be applied, in what order. It looks for the `$EDITOR` environment variable, and if it can't find it, tries `pico`. If both of those fail, the script exits. === More options === * `-t <option>`, `--test <option>`: `none`, `files`, `directory`, `long` This determines which files to doctest after applying the patches. `none` will doctest nothing, `files` will doctest all files touched by the patches, `directory` will doctest all directories containing a file that was changed, and `long` will call `make ptestlong`. * `-f`, `--finish`: On completion of testing, if all tests succeeded, commit all changes to all repositories. Incompatible with `-l`. * `-l`, `--leave-in-queue`: On the completion of testing, leave all patches in queues (whether tests failed or succeeded), as long as all patches applied correctly. Incompatible with `-f`. * `-n <number>`, `--num-threads <number>`: number of threads to use for parallel doctesting. * `-r <repo>`, `--repository <repo>`: default repository to commit patches to * `-d <dir>`, `--directory <dir>`: directory to store patch files in (defaults to a temp dir) * `-o`, `--overwrite`: if storing files to a different directory, whether or not to overwrite files when downloading === Standard use cases === * `sage -merge <ticket_number> -t long -n 12`: This will take a ticket, merge patches from that ticket, and run long doctests on the whole library with 12 threads. This is a very common case. Adding `-l` will let you inspect the patches and commit afterwards, and `-f` will simply commit if everything worked. * `sage -merge -a -f -t long`: This basically creates a new alpha release. `:)` It merges patches with positive review from trac, running long doctests after each, and committing those that pass all doctests. === sage -merge todo list === * `sage -merge` seems to fail when run from certain directories (such as inside the sage tree). Look into this and fix it -- likely just add an `os.chdir` somewhere at the start of the script (maybe to `$SAGE_ROOT`?). * It would be very easy to make `sage -merge -c` take an optional argument to show tickets which need review, or tickets that need work. (In fact, `all_tickets` takes an argument called `report` for exactly this reason.) * Add code to test coverage of the code after the patches are applied. * Add automatic logging to a file, with the default name including the ticket number, and a command line option to change it. * Currently, one can't use this script if there are patches sitting in the queue. This wouldn't be too hard to add -- one needs to use the `hg qsave` and `hg qrestore` options to keep track of '''where''' to move back to in the queue. See `hg help qsave` for more info on how this works. * Add more options for which tests to run -- in particular, something for optional doctests is needed. * The list of what files to doctest (for the choice of doctesting everything) is currently hard-coded, and was copied out of the top-level makefile for Sage. This shouldn't be hard-coded. * Make the warnings from the docbuilding process turn into errors, so that no new patches which cause the docs not to build will get merged unnoticed. * Add an option to use pylint, pyflakes, pychecker, and possibly other tools to the new code. (Since the sage code base doesn't currently have this, one might need to call it on the files getting changed before they're patched, then call it again after, and diff the two.) * Make an option for `sage -merge -a` to intelligently apply patches in sequence, so that one can see what goes on with applying '''all''' the patches at once (not just one at a time). One can do this with `-f`, but that will commit things to your repository. (Of course, you could use a clone and throw it away if it fails ...) * Make better options for allowing people to apply patches from the same ticket to different repositories (i.e. one to `sage-main`, one to `sage-scripts`, etc.). This could involve coming up with a good way to notate this when editing the list of patches. Or, one could try each patch on every repository until it works on one, assuming that no patch would cleanly apply to multiple repositories. * Add a system to make default choices for which patches to merge, so that no interaction from the user is needed. (This would make batch processing all positive reviewed patches possible.) * Display the trac file comments as comments after patches. For example, {{{ http://trac.sagemath.org/sage_trac/raw-attachment/ticket/6273/trac_6273.patch http://trac.sagemath.org/sage_trac/raw-attachment/ticket/6273/trac_6273-replacement.patch http://trac.sagemath.org/sage_trac/raw-attachment/ticket/6273/trac_6273_new.patch }}} should be {{{ http://trac.sagemath.org/sage_trac/raw-attachment/ticket/6273/trac_6273.patch http://trac.sagemath.org/sage_trac/raw-attachment/ticket/6273/trac_6273-replacement.patch http://trac.sagemath.org/sage_trac/raw-attachment/ticket/6273/trac_6273_new.patch # Replaces both previous }}} (also, strongly encourage people to add comments to files when uploading in trac) == Mandatory builds == Sage is regularly built and doctested on various machines within the Sage compute cluster and the !SkyNet cluster. The Sage compute cluster consists of the following machines: 1. `bsd.math` Mac OS X 1. `rh.math` Ubuntu 1. `[boxen,sage].math` Ubuntu 1. `t2.math` Solaris 10 The machine `boxen.math` hosts various Linux virtual machines for building Sage binaries. The [[skynet|SkyNet]] cluster consists of the following machines: 1. `cicero` Fedora 1. `cleo` Red Hat Enterprise Linux Server 1. `eno` Fedora 1. `flavius` Fedora 1. `fulvia` Solaris 10 1. `iras` SUSE 1. `lena` Fedora 1. `mark` Solaris 10 1. `mark`2 Solaris 10 1. `sextus2` Fedora 1. `taurus` Fedora Some contributors occasionally build/test Sage on machines within the [[http://gcc.gnu.org/wiki/CompileFarm|GCC compile farm]]. If you are a developer of a free software project, such as Sage, you are encouraged to write to the relevant system administrator of that compile farm to request an account on that farm. See the GCC compile farm's page for more information. The following lists a minimal set of machines on which Sage should be built and doctested prior to announcing an alpha or RC release to the sage-devel and sage-release mailing lists. 1. bsd.math --- Mac OS X 1. cleo --- Red Hat 1. eno --- Fedora 1. sage.math --- Ubuntu 1. t2.math --- Solaris Having built and doctested Sage on the above machines, you can then announce an alpha/RC release to the relevant lists. People are then encouraged to build/test Sage on their own machines, and even build/test Sage on all machines within the Sage and !SkyNet clusters, and the GCC compile farm. == Doctesting on the sage.math cluster == When you run doctests with "make ptest" or "make ptestlong", Sage will by default use a number of threads equal to the number of cores. On sage.math, this is often inadvisable, so when running doctests, always use something like {{{make ptest NUM_THREADS=16}}} (replace 16 with a sensible number; 24 minus the current load average will usually be considered "sensible") == Building on boxen.math == The machine boxen.math hosts a number of virtual machines that run various 32- and 64-bit Linux distributions. Here is a list of supported Linux distributions on boxen.math: * CentOS 5.3 (32- and 64-bit) --- centos5.3-32 and centos5.3-64 * Debian 5.0 Lenny (32- and 64-bit) --- debian5-32 and debian5-64 * Fedora 11 (32- and 64-bit) --- fedora11-32 and fedora11-64 * Mandriva 2009.1 (32- and 64-bit) --- mandriva2009.1-32 and mandriva2009.1-64 * openSUSE 11.1 (32- and 64-bit) --- opensuse-11.1-32 and opensuse-11.1-64 * Ubuntu 9.04 (32- and 64-bit) --- ubuntu9.04-32 and ubuntu9.04-64 The following Windows XP virtual machines are also hosted on boxen.math: * winxp1 * winxp2 * winxp3 Before starting the build process on the build farm, first login to boxen.math using either one of two methods shown below: {{{ $ ssh -C -x -a <username>@boxen # use this syntax if you're # already login to the Sage cluster $ ssh -C -x -a <username>@boxen.math.washington.edu # this syntax is required if # you're not already login to the Sage cluster }}} Replace `<username>` with your username. Next, ensure that you have the privilege to run as the username `wstein`. If you are able to sudo as username `wstein`, then do so as follows: {{{ $ sudo -H -u wstein bash }}} The option `-H` would set your home directory to that of username `wstein`, the option `-u wstein` is the username you want to sudo as (in this case it's `wstein`), and `bash` would login with the Bash shell. Delete all source tarballs in `/home/wstein/farm/src` and place the source tarball you want to build in `/home/wstein/farm/src`. Navigate to the directory `/home/wstein/farm` and run `./go`, which would actually start the build process on the Linux virtualized guests. The results would appear in `/home/wstein/farm/out/<version_number>/` as they complete. You can watch the build progress on all virtualized guests using {{{ $ tail -f /home/wstein/farm/out/<version_number>/* }}} or watch the progress of a virtualized guest as follows: {{{ $ tail -f /home/wstein/farm/out/<version_number>/<guest_name>.out }}} Press Control-C to quit watching. Known issues: * Both of the 32- and 64-bit Mandriva machines would fail trying to build Python, because the Python spkg won't build under nohup. Start a screen session, ssh into both boxes, and restart it by hand -- or, better yet, diagnose and fix the problem. == Building on SkyNet == Use `/home/wstein/farm/go_skynet` to start the build on SkyNet. The files appear in `/home/wstein/build/logs/<version_number>`. As of 4.0.2, eno and cleo works fine, but the rest would fail. You can restart the builds on cicero and iras by hand, and they should complete just fine. == Releasing source tarballs and binaries == You can use `sage -sdist <version>` to create a new source tarball, and `sage -bdist <version>` to create a new binary. These scripts are very particular about the formatting of version strings, see trac #6338. Both `sage -sdist <version>` and `sage -bdist <version>` take care of updating the Sage banner for you so you don't need to manually edit the file `SAGE_ROOT/local/bin/sage-banner`. It's very useful to put the various bits of the release in /home/release on sage.math, and to follow a directory structure somewhat like this: http://sage.math.washington.edu/home/release/sage-4.5.2.alpha0 which includes a sage.math binary, source tarball, md5sums, a directory for upgrading, and the text of the release note. It's also nice to put Solaris/t2.math binaries there, too. Using a standard location for releases makes it easier for developers to follow things. '''NOTE''': When cutting a new rc or alpha release, make sure to '''always''' produce binaries for sage.math. This also applies to any stable release -- several people will make use of these. The download page for 64-bit Linux contains a text file called {{{sage-x.y.z-sage.math.washington.edu-x86_64-Linux.txt}}}. This file lists directories on sage.math where sage.math account holders are able to get a copy of a binary specifically built for that machine. '''NOTE''': After an sdist, one needs to call `sage -br` to rebuild the library. This is annoying, and should be fixed. == Release checklist == Here's a list of things you want to make sure to do before every release: * Check that sage starts (you can use `local/bin/sage-starts` or check by hand) * Check that `notebook()` works * Check that `notebook(secure=True)` works * Check that `wiki()` works * Check that the [[http://www.sagemath.org/doc/developer/sage_manuals.html#building-the-manuals | documentation builds]], both the HTML and PDF versions. * Check that `sage -startuptime` works, and isn't worse than the last release. * Check that `sage -upgrade` works, at least from the last release. * Make sure to [[ValgrindingSage|Valgrind Sage]] late in the release cycle if this is a major release. * Use `sage -coverage` to test that the coverage hasn't gone down. * If any patches add files to the sage library itself, the release manager should check that the rules in MANIFEST.in include these files, or they will be missing from sage-x.x.x.spkg. * When a new package is added to the standard spkg repository, this usually involve corresponding changes in the files `SAGE_ROOT/spkg/standard/deps` and `SAGE_ROOT/spkg/install`. The updated standard spkg repository needs to be copied over to the master server on sagemath.org. Keep in mind that the standard spkg repository contains the file `SAGE_ROOT/spkg/standard/deps` so copying that repository over to the master server automatically copies the `deps` dependencies file. However, the file `SAGE_ROOT/spkg/install` must be manually copied over to the master server so it can be mirrored out. * We should incorporate max timings into the automerge scripts. Maybe reject a ticket if it triples the testing time or something. It should also say "oh, and doctesting these files took more than twice as long as normal." * Double check the release note and release tour. * Generate a release note using [[http://sage.math.washington.edu/home/mvngu/apps/rnotes/ | this script]], which is also available at http://bitbucket.org/mvngu/rnotes/ * When releasing the source version of a stable release, send a release announcement to [[http://groups.google.com/group/sage-devel|sage-devel]] and [[http://groups.google.com/group/sage-release|sage-release]]. The release announcement should be accompanied with the release note. Upload the release note to http://www.sagemath.org/mirror/src/announce/ and prepend the HISTORY.txt file with that latest release note. This cumulative HISTORY.txt file should also be up on http://www.sagemath.org/mirror/src/announce/. * When releasing binaries for a particular stable version, send a release announcement to [[http://groups.google.com/group/sage-announce|sage-announce]]. That release announcement should be accompanied with the release note, and it should also contain a link to the release tour. * Notice, that the script "export" in ~/www-files/ in sagemath's home directory is capable of doing this automatically. it's input is the source tarball and the release notes file and it also updates the SPKGs and mirrors everything out. * Update the Mercurial repository at http://hg.sagemath.org == Optional Packages == If you don't know how, get a more senior release manager to do this part for you. * Log in as user sagemath on boxen: $ sudo -H -u sagemath bash * Update spkg in ~/www-files/packages/optional (the txt files are automatically updated, so don't worry about them). * Run $ ~/www-files/packages/gen_html * Copy the modified files to the master mirror: $ ~/mirror (you can check it before mirroring at sagemath.org/mirror/ ) * That's it! |
