= Documentation tools: Sphinx, ... = == Autodoc == Jeroen worked on autodoc and partially "unforked" it. The Sage-specific fixes were kept, but sage_autodoc is now much closer to upstream than before. See http://trac.sagemath.org/ticket/20359 == Upgrading to Sphinx 1.4 == Robert worked on generating Sage docs with native Sphinx. This should bring: * native support parallelism * removal of some hacks for Cython support * ... Sphinx 1.4 now mostly works, but the merging of the final index breaks badly: http://trac.sagemath.org/ticket/18497 == Getting rid of autogenerated .rst stub files == Goals: * simpler build system (no need to maintain essentially duplicate information) * more robust build w.r.t. deleting python files * cleaner, ... Robert started implementing the required features in Sphinx master == Multidoc == TODO: Florent: status report * Since 1.3, Sphinx can also read in parallel. * [robert] sphinx-build -j ${NUMBER_OF_SAGE_DOC_PACKAGES} should ideally have the same effect. * Original PR: https://bitbucket.org/birkenfeld/sphinx/pull-requests/108/wip-parallel-build-experimentation/diff == Analyzis of the memory footprint of Sphinx (Florent, Robert, Julien) == Analysis: * A large cache is kept in memory for autofunction and automethods. Since we only use automodule, it could be dropped once the docstring of the module is analysed (TODO Upstream). The cache is filled with all docstring encoded in UCS32. Some string are 3BM large. * Unpickling the environment and doctrees files takes anound 1.6GB that's what we might hope as a measure of peak memory consumption. * The parallel compiling doesn't seem to increase significantly the needed memory. == Jupyter notebook exporter for Sphinx == Robert pointed to preliminary work by Harald Schilly: https://github.com/sphinx-doc/sphinx/pull/2117 == To be discussed with Robert: == * Error reporting of Sphinx when documenting API. How to locate the error.? == Random notes == * Discussion on "inspect" I do not get because I do not know what "inspect" is * This is time consuming: each time one has to rebuild the whole documentation. * Jeroen is concerned about images in the docs and build times. == Notes from Robert (Sphinx Dev) == TODO: merge with the above? I have visited the [1]Sage Days 77 in Cernay, France, this week to help them with their Sphinx setup. Sage is looking at adopting vanilla Sphinx; autodoc will be a big upstreamer for them, memory might be a show stopper. [2]Sage is a distribution of mathematical Python libraries (or "mathematics software system"), currently bundling Sphinx 2.2.x. Their [3]documentation consists of ~200 plain text files and ~2000 [4]automodule stubs. This brings a number of [5]interesting challenges. First off, they have their own [6]copy of autodoc to support a number of use cases around Cython and wrapped functions. Since it diverged back around Sphinx 1.0, it is unclear what's required from our side. They have started [7]work to merge forward, and will then approach us with whatever extension points / improvements they'd need. Most of these fixes should be useful in core Sphinx as well. They also have their [8]own poor man's parallelization (splitting up the documentation into multiple Sphinx projects and then building through make -j) based on sphinx-multidoc. We have experimented with Sphinx's parallel build support, and it seems to give similiar speedups. Last but not least, they also have their own [9]copy of apidoc. We agreed that plain sphinx-apidoc covers them well, with the caveat that it'd be a big advantage for them (and other projects, I believe) if they could spare the stub automodule files. They suggested putting "automodule:" statements directly into the toctree, which I have a rough draft implementation of. Building their apidoc consumes 1.8G in memory for the doctrees alone. Their resource footprint peaks at 2.6G (or at 3+G with extensions enabled); doctrees are 200M in pickled format. I have filed some bugs with the results of our memory forensics. There are a couple of caches that unnecessarily blow up the memory consumption ([10]MemoryAnalyzer, [11]sys.modules, and probably linecache.) and quite a few opportunities to reduce the doctree in-memory blowup (see [12]#2426 for Text nodes.) I can only recommend other Sphinx developers (to be) to visit user projects like this. It is a valuable learning opportunity for us.