Cython part 2
system:sage


<h1>Cython</h1>
<address>Jeroen Demeyer (Universiteit Gent)</address>
<h2>except values</h2>
<p>When defining a <strong>cdef</strong> (of <strong>cpdef</strong>) function, you can explicitly define a return type:</p>

{{{id=1|
cython("""
cpdef long square(n):
    return n*n
""")
///
}}}

{{{id=4|
square(4)
///
}}}

{{{id=51|
type(square(4))
///
}}}

<p>Since C does not support Python exceptions, there is no way that this code can raise an exception. Any exceptions which are seen are just ignored:</p>

{{{id=6|
square("x")
///
}}}

<p>This can be solved using "except" values. This is a special return value which is interpreted as an exception:</p>

{{{id=5|
cython("""
cpdef long square(n) except -1:
    return n*n
""")
///
}}}

{{{id=9|
square("x")
///
}}}

<p>Of course, we now get an error if the return value is really -1:</p>

{{{id=10|
square(I)
///
}}}

<p>For this, you would use <strong>except?</strong> (if that value is returned, Cython checks whether the return value corresponds to an exception or not):</p>

{{{id=12|
cython("""
cpdef long square(n) except? -1:
    return n*n
""")
///
}}}

{{{id=15|
square(I)
///
}}}

{{{id=16|
square("x")
///
}}}

<p>For functions returning Python objects (it doesn't matter if it's a specific type or not), exceptions are handled by passing a NULL pointer, you do not need an except value in this case.</p>
<h2>Interrupts</h2>
<ul>
<li>This section is Sage-specific, upstream Cython has no support for interrupting.</li>
<li>In the Sage library, you need to write <strong>include "sage/ext/interrupt.pxi"</strong> in your .pyx file (not the .pxd file!) to use these functions.</li>
</ul>
<h3>sig_check</h3>
<p>We already saw <strong>sig_check()</strong>, which checks whether we have "missed" an interrupt. If an interrupt signal (SIGINT) is produced (typically by pressing CTRL-C), the interrupt framework sets an internal flag<strong>. sig_check()</strong> just looks at this flag and raises <strong>KeyboardInterrupt</strong> (which can be handled as usual).</p>

{{{id=29|
cython("""
include "sage/ext/interrupt.pxi"
def sum_of_integers(unsigned long n):
    cdef unsigned long total = 0
    cdef unsigned long i
    for i in range(n+1):
        sig_check()
        total += i
    return total
""")
///
}}}

{{{id=52|
sum_of_integers(10)
///
}}}

{{{id=31|
sum_of_integers(10^18)
///
}}}

<p><strong>sig_check()</strong> is fast and safe, so remember to use it when you write new Cython code in Sage.</p>
<h3>sig_on / sig_off</h3>
<p>When your code just calls an external library, <strong>sig_check()</strong> is not an option.</p>

{{{id=27|
cython("""
from sage.libs.gmp.mpz cimport *
from sage.rings.integer cimport Integer
def binomial(unsigned long n, unsigned long k):
    cdef Integer r = Integer.__new__(Integer)
    mpz_bin_uiui(r.value, n, k)
    return r
""")
///
}}}

{{{id=32|
%time
try:
    alarm(0.1)
    _ = binomial(10^8, 10^7)
except KeyboardInterrupt:
    pass
///
}}}

<p>To solve this use case, you can use <strong>sig_on()</strong> and<strong> sig_off()</strong>.</p>

{{{id=33|
cython("""
from sage.libs.gmp.mpz cimport *
from sage.rings.integer cimport Integer
def binomial(unsigned long n, unsigned long k):
    cdef Integer r = Integer.__new__(Integer)
    sig_on()
    mpz_bin_uiui(r.value, n, k)
    sig_off()
    return r
""")
///
}}}

{{{id=37|
%time
try:
    alarm(0.1)
    _ = binomial(10^8, 10^7)
except KeyboardInterrupt:
    pass
///
}}}

<h3>What does sig_on() do?</h3>
<ul>
<li>When an interrupt signal is received, the running code is <em>immediately</em> interrupted.</li>
<li>In particular, allocated memory is not freed, the internal state of objects is undefined.</li>
<li>The <strong>sig_on()</strong> statement raises a <strong>KeyboardInterrupt</strong> (so, if you want to catch the exception, the <strong>sig_on()</strong> statement must be <em>inside</em> the <strong>try</strong> block)</li>
</ul>
<h3>Warning!</h3>
<ul>
<li>Because <strong>sig_on()</strong> can mess up memory, you should avoid any Python code inside <strong>sig_on()</strong>/<strong>sig_off()</strong>.</li>
<li>After <strong>sig_on()</strong>, you must call <strong>sig_off()</strong> (beware in particular if your code has multiple exit points). When in doubt, use <strong>try</strong>/<strong>finally: sig_off()</strong> but in this case, you must put <strong>sig_on()</strong> <em>outside</em> this try block.</li>
</ul>
<p>For more information about how to use <strong>sig_on()</strong>, see <a href="http://www.sagemath.org/doc/developer/coding_in_cython.html#interrupt-and-signal-handling">Interrupt and Signal Handling</a> in the developer manual.</p>
<h2>More general signal and error handling</h2>
<p>Let's consider the example from yesterday which crashed Sage and wrap it inside <strong>sig_on()</strong>/<strong>sig_off()</strong>:</p>

{{{id=44|
cython("""
cimport cython
@cython.cdivision(True)
def trialdivision(unsigned long long n):
    cdef unsigned long long i = 0  # MISTAKE!
    sig_on()
    try:
        while i*i <= n:
            if n % i == 0:
                return i
            i = i+1
    finally:
        sig_off()
""")
///
}}}

{{{id=34|
trialdivision(10^7-1)
///
}}}

<p>As you can see,&nbsp;<strong>sig_on()</strong> also handles segfaults (and other fatal signals like Floating point exception in this example) and turns them into ordinary Python exceptions.</p>

<p>Some C libraries also use the <strong>sig_on()</strong> mechanism to pass errors. In particular, this happens with memory allocation errors in MPIR:</p>

{{{id=35|
binomial(10^18, 10^17)
///
}}}

<p>This is implemented by calling <strong>sig_error()</strong> in some callback function:</p>

{{{id=48|
cython("""
from cpython.exc cimport PyErr_SetString

cdef library_callback():
    PyErr_SetString(RuntimeError, "something went wrong!")
    sig_error()

sig_on()
library_callback()
sig_off()
""")
///
}}}

<h2>Safe memory allocations</h2>
<ul>
<li>In Cython code, do not use the standard <strong>malloc</strong> (or<strong> realloc</strong>,<strong> calloc</strong>,<strong> free</strong>) functions.</li>
<li>Instead, use&nbsp;<strong>sage_malloc</strong>: this behaves exactly like <strong>malloc</strong> but it's safer because no interrupts can happen during a <strong>malloc()</strong> call (those would be very bad!).</li>
<li>There is also <strong>check_malloc</strong>, <strong>check_realloc</strong> and <strong>check_calloc</strong> which raise <strong>MemoryError</strong> instead of returning NULL.</li>
<li>For allocating arrays, there are BSD-inspired <strong>check_allocarray</strong> and <strong>check_reallocarray</strong> which avoids overflow in a call like <strong>malloc(n * sizeof(type))</strong>.</li>
<li><strong>try</strong>/<strong>finally</strong> is your friend to avoid memory leaks (put <strong>sage_free</strong> inside the <strong>finally</strong>).</li>
</ul>
<p>The <strong>sage_</strong> functions are made available by <strong>include "sage/ext/stdsage.pxi"</strong>, the <strong>check_</strong> functions must be cimported from <strong>sage.ext.memory</strong>.</p>
<h2>Interfacing a C library</h2>
<h3>Step 1: building the library</h3>
<p>Obviously, the first step is to build your library in Sage:</p>
<ul>
<li>Create the <strong>build/pkgs/&lt;packagename&gt;</strong> directory and fill it with the needed files.</li>
<li>Run <strong>./sage -i &lt;packagename&gt;</strong> and hope for the best...<strong><br /></strong></li>
<li>For a standard package: add a <strong>newest_version</strong> line to <strong>build/install</strong> and add the package to <strong>build/deps</strong>. Run <strong>make distclean &amp;&amp; make</strong> to check that it works.</li>
</ul>
<h3>Step 2: creating .pxd file(s)</h3>
<p>To create a Cython interface, you need to add one (or more) .pxd file(s) with the declarations of all functions. As an example, let's look at <a href="https://github.com/sagemath/sage/tree/develop/src/sage/libs/flint">src/sage/libs/flint</a>.</p>
<p>All declarations are in <strong>cdef extern from "header.h"</strong> blocks. This has several consequences:</p>
<ul>
<li>Cython will not complain that you didn't implement the function.</li>
<li>Cython will generate an <strong>#include "header.h"</strong> line in the C source.</li>
<li>Cython will add <strong>header.h</strong> as dependency: distutils will recompile the C file if <strong>header.h</strong> changed.</li>
</ul>
<p>For C structs, it is not needed to give all members. Only the members which need to be accessed by Cython need to be given.</p>
<p>The Cython distribution contains .pxd files for <strong>cpython</strong> (Python API functions), <strong>libc</strong> (C standard library), <strong>libcpp</strong> (C++ standard library) <strong>posix</strong> (POSIX system calls), <strong>numpy</strong> and <strong>openmp</strong>, so you can just cimport those functions.<strong></strong></p>
<h3>Step 3: writing code</h3>
<p>Implement the Sage functionality you want in a new .pyx file. If you want to cimport your new cdef functions/classes in other modules, then also create a corresponding .pxd file.</p>
<h3>Step 4: edit module_list.py</h3>
<p>To ensure your new module will be compiled, you need to add it to <a href="https://github.com/sagemath/sage/blob/develop/src/module_list.py">module_list.py</a>. Be sure to add a <strong>libraries=['my_cool_lib']</strong> argument to your <strong>Extension</strong>.</p>
<p>Now, just run <strong>./sage -b</strong> and your module should be compiled and become available in Sage.</p>

{{{id=50|

///
}}}