Cython

Jeroen Demeyer (Universiteit Gent)

except values

When defining a cdef (of cpdef) function, you can explicitly define a return type:

{{{id=1| cython(""" cpdef long square(n): return n*n """) /// }}} {{{id=4| square(4) /// }}} {{{id=51| type(square(4)) /// }}}

Since C does not support Python exceptions, there is no way that this code can raise an exception. Any exceptions which are seen are just ignored:

{{{id=6| square("x") /// }}}

This can be solved using "except" values. This is a special return value which is interpreted as an exception:

{{{id=5| cython(""" cpdef long square(n) except -1: return n*n """) /// }}} {{{id=9| square("x") /// }}}

Of course, we now get an error if the return value is really -1:

{{{id=10| square(I) /// }}}

For this, you would use except? (if that value is returned, Cython checks whether the return value corresponds to an exception or not):

{{{id=12| cython(""" cpdef long square(n) except? -1: return n*n """) /// }}} {{{id=15| square(I) /// }}} {{{id=16| square("x") /// }}}

For functions returning Python objects (it doesn't matter if it's a specific type or not), exceptions are handled by passing a NULL pointer, you do not need an except value in this case.

Interrupts

This section is Sage-specific, upstream Cython has no support for interrupting.
In the Sage library, you need to write include "sage/ext/interrupt.pxi" in your .pyx file (not the .pxd file!) to use these functions.

sig_check

We already saw sig_check(), which checks whether we have "missed" an interrupt. If an interrupt signal (SIGINT) is produced (typically by pressing CTRL-C), the interrupt framework sets an internal flag. sig_check() just looks at this flag and raises KeyboardInterrupt (which can be handled as usual).

{{{id=29| cython(""" include "sage/ext/interrupt.pxi" def sum_of_integers(unsigned long n): cdef unsigned long total = 0 cdef unsigned long i for i in range(n+1): sig_check() total += i return total """) /// }}} {{{id=52| sum_of_integers(10) /// }}} {{{id=31| sum_of_integers(10^18) /// }}}

sig_check() is fast and safe, so remember to use it when you write new Cython code in Sage.

sig_on / sig_off

When your code just calls an external library, sig_check() is not an option.

{{{id=27| cython(""" from sage.libs.gmp.mpz cimport * from sage.rings.integer cimport Integer def binomial(unsigned long n, unsigned long k): cdef Integer r = Integer.__new__(Integer) mpz_bin_uiui(r.value, n, k) return r """) /// }}} {{{id=32| %time try: alarm(0.1) _ = binomial(10^8, 10^7) except KeyboardInterrupt: pass /// }}}

To solve this use case, you can use sig_on() and sig_off().

{{{id=33| cython(""" from sage.libs.gmp.mpz cimport * from sage.rings.integer cimport Integer def binomial(unsigned long n, unsigned long k): cdef Integer r = Integer.__new__(Integer) sig_on() mpz_bin_uiui(r.value, n, k) sig_off() return r """) /// }}} {{{id=37| %time try: alarm(0.1) _ = binomial(10^8, 10^7) except KeyboardInterrupt: pass /// }}}

What does sig_on() do?

When an interrupt signal is received, the running code is immediately interrupted.
In particular, allocated memory is not freed, the internal state of objects is undefined.
The sig_on() statement raises a KeyboardInterrupt (so, if you want to catch the exception, the sig_on() statement must be inside the try block)

Warning!

Because sig_on() can mess up memory, you should avoid any Python code inside sig_on()/sig_off().
After sig_on(), you must call sig_off() (beware in particular if your code has multiple exit points). When in doubt, use try/finally: sig_off() but in this case, you must put sig_on() outside this try block.

For more information about how to use sig_on(), see Interrupt and Signal Handling in the developer manual.

More general signal and error handling

Let's consider the example from yesterday which crashed Sage and wrap it inside sig_on()/sig_off():

{{{id=44| cython(""" cimport cython @cython.cdivision(True) def trialdivision(unsigned long long n): cdef unsigned long long i = 0 # MISTAKE! sig_on() try: while i*i <= n: if n % i == 0: return i i = i+1 finally: sig_off() """) /// }}} {{{id=34| trialdivision(10^7-1) /// }}}

As you can see, sig_on() also handles segfaults (and other fatal signals like Floating point exception in this example) and turns them into ordinary Python exceptions.

Some C libraries also use the sig_on() mechanism to pass errors. In particular, this happens with memory allocation errors in MPIR:

{{{id=35| binomial(10^18, 10^17) /// }}}

This is implemented by calling sig_error() in some callback function:

{{{id=48| cython(""" from cpython.exc cimport PyErr_SetString cdef library_callback(): PyErr_SetString(RuntimeError, "something went wrong!") sig_error() sig_on() library_callback() sig_off() """) /// }}}

Safe memory allocations

In Cython code, do not use the standard malloc (or realloc, calloc, free) functions.
Instead, use sage_malloc: this behaves exactly like malloc but it's safer because no interrupts can happen during a malloc() call (those would be very bad!).
There is also check_malloc, check_realloc and check_calloc which raise MemoryError instead of returning NULL.
For allocating arrays, there are BSD-inspired check_allocarray and check_reallocarray which avoids overflow in a call like malloc(n * sizeof(type)).
try/finally is your friend to avoid memory leaks (put sage_free inside the finally).

The sage_ functions are made available by include "sage/ext/stdsage.pxi", the check_ functions must be cimported from sage.ext.memory.

Interfacing a C library

Step 1: building the library

Obviously, the first step is to build your library in Sage:

Create the build/pkgs/<packagename> directory and fill it with the needed files.
Run ./sage -i <packagename> and hope for the best...
For a standard package: add a newest_version line to build/install and add the package to build/deps. Run make distclean && make to check that it works.

Step 2: creating .pxd file(s)

To create a Cython interface, you need to add one (or more) .pxd file(s) with the declarations of all functions. As an example, let's look at src/sage/libs/flint.

All declarations are in cdef extern from "header.h" blocks. This has several consequences:

Cython will not complain that you didn't implement the function.
Cython will generate an #include "header.h" line in the C source.
Cython will add header.h as dependency: distutils will recompile the C file if header.h changed.

For C structs, it is not needed to give all members. Only the members which need to be accessed by Cython need to be given.

The Cython distribution contains .pxd files for cpython (Python API functions), libc (C standard library), libcpp (C++ standard library) posix (POSIX system calls), numpy and openmp, so you can just cimport those functions.

Step 3: writing code

Implement the Sage functionality you want in a new .pyx file. If you want to cimport your new cdef functions/classes in other modules, then also create a corresponding .pxd file.

Step 4: edit module_list.py

To ensure your new module will be compiled, you need to add it to module_list.py. Be sure to add a libraries=['my_cool_lib'] argument to your Extension.

Now, just run ./sage -b and your module should be compiled and become available in Sage.

{{{id=50| /// }}}