On development models for sharing (experimental) code

One core aim of Sage is to foster code sharing, and to encourage groups of researchers, teachers, and other users to get together to develop new features they need either on top of or within Sage, and share them.

Over the years, many development workflows have been experimented by various groups of people to improve Sage in certain areas, like Sage-Combinat for (algebraic) combinatorics, Sage-Words for combinatorics on words, SageManifolds for differential geometry, purple-sage for number theory, ...

The goal of this document is to discuss the different workflows that have been tried with their pros and cons, to share best practices and to brainstorm about what support and recommendations Sage could provide for various use cases. Eventually, this could become a section of the developer's manual (though this can be of interest for other people wanting to start sharing code without necessarily contributing to Sage), or a page of the sagemath.org website.

At this point this is a collection of notes by N. ThiƩry; please hack in and contribute your own vision!

See also:

Objectives of a development workflow

Of course the milleage will vary from project to project, but the objectives of a development workflow can typically be to:

1. Support fast paced development within a group of users working on the same topic, or needing similar features.

2. Support rapid dissemination of experimental features.

3. Foster high quality code by promoting documentation, tests, code reviews.

4. Foster intrinsic high quality code by providing an *ecosystem* where (experimental) code can live, compete with other implementations, mature and be selected or die, all at a fine granularity.

5. Strike a balance between centralized and decentralized.

6. Minimize *maintenance* overhead, and in particular code rotting.

7. Remain flexible between the all-in-one versus packages development models (simplifying things out: between Sage's model and GAP's model).

8. Promote extending existing Sage classes and modules with additional features.

Existing workflows

Direct integration into Sage

In this workflow, each feature is shared by integrating it directly into Sage.

Pros:

Cons:

Examples:

Discussion:

Feature branches

In this workflow, features or feature sets are implemented as branches on the Sage sources.

Pros:

Cons:

Examples:

Features in trac:

Sage development trac can be used to host the features. The ticket for a feature has milestone sage-feature and keeps a feature branch that provides a special functionality that can be merged to the Sage core by the user. The ticket is not reviewed (i.e, not goes into needs review status) until it is accepted as a standard feature and integrated to Sage. The user can select a set of features at build time and pull the feature branches from trac and start to make in the usual way. A special script "make-sage-with features" might be provided to make this easy.

A feature in trac should provide at least

Other remarks:

Patch queue as used by Sage-Combinat between 2009 and 2013

See also the bottom of this page.

TODO: description

This section is just for reference: there used to be a strong rationale for this workflow with the former Sage development workflow and a given context. But not any more.

Pros:

Cons:

Standalone (pip) packages

Here the idea is to implement feature sets as independent Python packages on top of Sage. Converting a bunch of Python files into such a package to make it `easy to install <http://python-packaging-user-guide.readthedocs.io/en/latest/distributing/>`_ is straightforward.

Examples:

NON-Examples:

Pros:

Cons:

Standalone (pip) packages with an integration mission

This is a variant on the previous development workflow, with an explicit focus on easing (or even promoting) the integration of mature code into Sage.

Specifics:

Examples:

Pros:

Cons:

=== Standalone (pip) packages that provide native namespace packages ==

This is another variant, enabled by the transition to Python 3.

See https://trac.sagemath.org/ticket/28925

Specifics:

Examples:

Historical: what was this Sage-Combinat queue madness about???

Sage-Combinat is a software project whose mission is: "to improve the open source mathematical system Sage as an extensible toolbox for computer exploration in (algebraic) combinatorics, and foster code sharing between researchers in this area".

In practice it's a community of a dozen regular contributors, 20 occasional ones and, maybe, 30 users. They originally collaborated together on a collection of experimental patches (i.e. extensions) on top of Sage. Each one describes a relatively atomic modification which may span several files; it may fix a bug, implement a new feature, improve some documentation. The intent is that most of those extensions get integrated into Sage as soon as they are mature enough, with a typical life-cycle ranging from a few days to a couple months. In average 20 extensions are merged in each version of Sage (42 in Sage 5.0!), and more than 200 are under development.

Why do we want to share our experimental code

Here are our goals in using the Sage-Combinat queue for sharing patches:

== What are our constraints ==

Some random questions

Foreseeable future