On development workflows for sharing (experimental) code

One core aim of Sage is to foster code sharing, and encourage groups of researchers, teachers, and other users to get together to develop new features they need either on top of or within Sage, and share them.

Over the years, many development workflows have been experimented by various groups of people to improve Sage in certain areas, like Sage-Combinat for (algebraic) combinatorics, Sage-Words for combinatorics on words, Sage-manifolds for differential geometry, purple-sage for number theory, ...

The goal of this document is to discuss the different workflow that have been tried, their pros and cons, to share best practices and brainstorm about what support and recommendations Sage could provide for various use cases.

At this point it's a collection of notes by N. ThiƩry; please hack in and contribute your own vision!

Objectives of a development workflow

Specifically, the objectives of a development workflow can be to:

1. Support *fast paced development* within a group of users working on

2. Support *rapid dissemination of experimental features*

3. Foster high quality code by promoting documentation, tests, code reviews

4. Foster intrinsic high quality code by providing an *ecosystem*

5. Strike a balance between centralized and decentralized.

6. Minimize *maintenance* overhead, and in particular code rotting

7. Remain flexible between the all-in-one versus packages development models

8. Promote extending existing Sage classes and modules with additional features.

Existing workflows

Direct integration into Sage

In this workflow, each feature is shared by integrating it directly into Sage.

Pros:

- Simplicity for the user: all stable features are directly available in Sage - Simplicity for Sage developers: no additional workflow to learn, no

- Promote early integration of code and 3. - 8. is straightforward

Cons:

- Limited support for 2. - Slows down the development: once a feature is in Sage, any change

- Getting the latest feature forces updating to the latest version of Sage - Introduces a bias toward code bloat (in doubt, features tend to be added to Sage)

Experimental feature branches

Pros:

- 8. is straightforward - Encourages integration into Sage - Development history is automatically kept upon integration into Sage

Cons:

- Branch needs to be regularly updated to prevent code rotting due to

- 2. requires basic git knowledge from end-users. - Lack of modularity for 2.: due to potential conflicts, it's not easy

- Cherry picking certain mature features for integration in Sage is

- It's hard to strike the right granularity in terms of feature /

- Because of the above, this workflow does not work well for 4. - Introduces a bias toward the all-in-one development model.

Patch queue as used by Sage-Combinat between 2009 and 2013

See also the following design notes about the Sage-Combinat workflow.

TODO: description

Pros:

- Relatively good for 1. (except for 6.) - Relatively good for 2. (thanks to "sage -combinat install"), except

- 8. is straightforward

Cons:

- Complexity of working at the meta level (version control on the patches) - Really bad at 6: Horrible maintenance overhead due to syntactic conflicts and lack of automatic merging - Introduces a strong bias toward code death, or at least non integration into Sage - Monolithic: one could not use several patch queues at once, so this

Using (pip) packages

Here the idea is to implement feature sets as independent Python packages on top of Sage. Converting a bunch of Python files into such a package, and making it easy to install is easy with e.g. pip:

https://python-packaging.readthedocs.io/en/latest/minimal.html

Pros: - Good for 1., 2., 4.,

Cons: - Handling of compatibility with various versions of the dependencies (in particular Sage) - Risk of code rotting (as Sage evolves over time) or death (if it's not maintained)

Using (pip) packages with an integration mission

This is a variant on the previous development workflow, with an explicit focus on easing (or even promoting) the integration of mature code into Sage.

Specific steps:

- layout the code as in the Sage library, with top module called

Pros: - Same as above - 8. is straightforward - Lighter maintenance overhead compared to branches or patch queues:

- The integration of mature code into Sage helps for 3 and for the

- Depending on how strongly one pushes toward the integration of