Writing Extension Modules
to be Interruptible

Zack Weinberg

〈zack@millionconcepts.com〉

〈zack@owlfolio.org〉

https://www.owlfolio.org/

If you’re reading this slide deck on the web, press S to bring up my notes.

A Common Bug

>>> import random
>>> def f(n=100000000):
...     return [random.random()
...         for _ in range(n)]
...
>>> f()
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in f
KeyboardInterrupt
>>>

< 0.1 s

>>> import numpy as np
>>> rng = np.random.default_rng()
>>> def g(n=1000000000):
...     return rng.random(n)
...
>>> g()
^CTraceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in g
KeyboardInterrupt
>>>

3.5 s

Here’s two versions of the same function. It generates a whole lot of random numbers. The version on the left uses the Python stdlib to generate them one at a time. On the right, instead, we generate them all at once with NumPy.

We run them both at the same time…

… and immediately realize that we forgot to store the result in a variable, so we hit control-C …

… we get the prompt back immediately for the pure Python version, but we have to wait a bit for the NumPy version.

On my computer, it’s a few tenths of a second versus three and a half full seconds.

It’s really easy to trip over this if you’re doing interactive number crunching.

What went wrong

NumPy didn’t call PyErr_CheckSignals
Common bug in compiled-code extensions
This talk is about:
1. Why extensions need to do that
2. Why, today, many extensions don’t do that
3. How we can make it easier to do that

Very short version of what went wrong: NumPy never bothered to call PyErr_CheckSignals. I’m not here to pick on NumPy specifically, lots of compiled code extensions have the same bug. But it is a bug.

The long version, now? That’s the rest of this talk.
Why is this something that you have to do?
Why don’t extensions do it already?
How can we make it easier for extensions to do the right thing here?

Audience background check

Raise your hand if…

you’ve written code in a compiled language
- C or assembly, specifically
you’ve written a compiled-code extension for CPython
- using the C-API directly, no helper libraries
you’ve written a multithreaded program
you’ve written a signal handler
you know the difference between thread-safe and async-signal-safe

What `Control-C` Does¹

Starts out like any keystroke
Converted to a signal, SIGINT
SIGINT delivered to Python interpreter
Python interpreter reacts by raising KeyboardInterrupt

¹ Except on Windows

Control-C is the keystroke that tells the Python interpreter—and lots of other programs—that they should stop doing what they’re doing and give control back to an interactive command loop. It’s at least a decade older than Python, and part of how it works is built into the Unix kernel.

Footnote, this and the next several slides are “how control-C works” on every operating system of interest except Windows. I don’t know what Windows does instead, I just know it’s different. But the good news is, somehow CPython and the C runtime library for Windows manage to paper over those differences; “you need to call PyErr_CheckSignals” is still the advice for extension authors on Windows.

So anyway, control-C starts out like any other keystroke, and we’re gonna skip over all the steps in between the keyboard and the “terminal line discipline” that’s the last stage of operating system processing before the CPython process gets some input. The line discipline sees control-C as ASCII control character hex 03 and, instead of passing it along to the Python process like regular input, it converts it to a thing called a signal and delivers that to the process. And the interpreter reacts to that signal by raising KeyboardInterrupt.

These words here. The signal is “delivered” and the interpreter “reacts.” The delay we saw earlier is in between these words. To understand why compiled-code extensions get in the way of the interpreter reacting immediately, we need to understand exactly what these words mean.

Signal Delivery

>>> import random
>>> def f(n=100000000):
...     return [random.random()
...         for _ in range(n)]
...
>>> f()
^C
Program received signal SIGINT
pymalloc_pool_extend ()
    at Objects/obmalloc.c:1361
(gdb) signal SIGINT
Continuing with signal SIGINT.
Breakpoint 2, signal_handler (sig_num=2)
    at ./Modules/signalmodule.c:347
(gdb)

>>> import numpy as np
>>> rng = np.random.default_rng()
>>> def g(n=1000000000):
...     return rng.random(n)
...
>>> g()
^C
Program received signal SIGINT
random_standard_uniform_fill ()
    at 0x00007ffff00b23a7
(gdb) signal SIGINT
Continuing with signal SIGINT.
Breakpoint 2, signal_handler (sig_num=2)
    at ./Modules/signalmodule.c:347
(gdb)

Here’s those two versions of the random number generation function again. We’re gonna run them again, and we’re gonna interrupt them again…

but this time, I’ve hooked up CPython to the C-level debugger, GDB. It intercepts the actual signal, and it gives you a chance to do something about it before the program ever sees it.

Side note, GDB is a very talkative program. I’ve cut a lot of unimportant detail out of the transcript on these slides. If you try this yourself, it’ll look differently. Also I set some breakpoints in advance, which we’ll see in effect…

when we tell GDB to let the signal go through to Python, and we immediately get told that we’re now about to execute Python’s signal handler.

The signal has been delivered at this point, and the interpreter could, in principle react right now… but it’s not going to. The signal_handler function is just going to set a flag, and the actual reaction—raising KeyboardInterrupt—won’t happen until we get back to the main interpreter loop.

That might seem like an unnecessary indirection, and it is the design-level cause of the delay due to NumPy in the code on the right, but there’s a good reason for it.

Signal Delivery

Breakpoint 2, signal_handler (sig_num=2)
    at ./Modules/signalmodule.c:347
(gdb) backtrace
#0  signal_handler (sig_num=2)
#1  <signal handler called>
#2  pymalloc_pool_extend (…)
#3  pymalloc_alloc (…)
#4  _PyObject_Malloc (…)
#5  _PyLong_FromMedium (…)
#6  PyLong_FromLong (…)
#7  _PyEval_EvalFrameDefault (…)
#8  PyEval_EvalCode (…)

(etc. etc.)

Breakpoint 2, signal_handler (sig_num=2)
    at ./Modules/signalmodule.c:347
(gdb) backtrace
#0  signal_handler (sig_num=2)
#1  <signal handler called>
#2  random_standard_uniform_fill (…)
#3  __pyx_f_5numpy_6random_7_common_f… (…)
#4  __pyx_pw_5numpy_6random_10_genera… (…)
#5  method_vectorcall_FASTCALL_KEYWORDS (…)
#6  _PyObject_VectorcallTstate (…)
#7  PyObject_Vectorcall (…)
#8  _PyEval_EvalFrameDefault (…)
#9  PyEval_EvalCode (…)

To understand why there’s this indirection, let’s look at the C-level traceback for both versions of the code, while we’re stopped at this breakpoint. GDB prints tracebacks the other way around from Python—the innermost frame is on top.

There’s a lot in there, let’s focus on the innermost three stack frames. Innermost is CPython’s signal_handler function, of course, but right above that we have this special “signal handler called” frame, and … huh. Frame two is the code we were executing when the signal arrived. On the right we have NumPy’s “fill vector with random numbers” code, and on the left, that looks like we’re allocating memory.

Signal delivery, y’see, is not like raising an exception. C doesn’t have exceptions. Signals don’t cancel execution of whatever they were doing when they arrive; they interrupt it. When signal_handler is done, the interpreter is going to go back to whatever it was doing when it was interrupted.

How did we get here???

(gdb) backtrace
#0  signal_handler (sig_num=2)
#1  <signal handler called>
#2  pymalloc_pool_extend (…)
...

(gdb) up
#1  <signal handler called>
(gdb) disassemble
Assembly dump of __restore_rt:
=> <+0>:    mov  rax, SYS_rt_sigreturn
   <+7>:    syscall
(gdb) up
#2  pymalloc_pool_extend (…)
(gdb) disassemble
Assembly dump of pymalloc_pool_extend:
...
   <+473>:  mov  dword [rdx+40], ecx
=> <+476>:  mov  qword [rdi], 0
...

I need to make you feel this so I’m going to show you the assembly language involved. Just for the pure-Python version, it’s the same principles for the other one. signal_handler is just a regular C function, but if we go up a frame…

…and then ask for disassembly…

…huh, that’s a system call stub, and a strange one: there’s no more code after the syscall instruction. Normally there should at least be a return instruction after the syscall. And if you grep the entire CPython source code for dunder restore_rt, you won’t find it at all.

And it gets weirder if we go up another level. This isn’t a function call sequence! GDB thinks that at this level we’re gonna execute the instruction marked with the fat arrow next, but the instruction before that isn’t a call instruction.

The black magic of signal delivery

Kernel “preempted” execution of pymalloc_pool_extend
Faked two stack frames to make CPU behave as if pymalloc_pool_extend called __restore_rt which called signal_handler
Resumed execution at beginning of signal_handler

(gdb) up
#1  <signal handler called>
(gdb) disassemble
Assembly dump of __restore_rt:
=> <+0>:    mov  rax, SYS_rt_sigreturn
   <+7>:    syscall
(gdb) up
#2  pymalloc_pool_extend (…)
(gdb) disassemble
Assembly dump of pymalloc_pool_extend:
...
   <+473>:  mov  dword [rdx+40], ecx
=> <+476>:  mov  qword [rdi], 0
...

This is the black magic of signal delivery.

The operating system kernel “preempted” execution of CPython at a completely arbitrary point in the middle of pymalloc_pool_extend. It can do this in between any pair of machine instructions!

Then it messed with the saved register state, and wrote two fake stack frames to CPython’s machine stack, to make the CPU behave as if this arbitrary point in pymalloc_pool_extend had called dunder restore_rt, which in turn called signal_handler. And then it resumed execution of CPython—at the first instruction of signal_handler, not where it was.

And this is why CPython needs to have that indirection, where signal_handler just sets a flag, and all further processing is postponed till we get back to the main interpreter loop. It’s not safe to raise a Python-level exception from this arbitrary point deep in the guts of pymalloc. Or that other arbitrary point deep in the guts of NumPy, or whatever other compiled code extension.

What `Control-C` Does¹

Starts out like any keystroke
Converted to a signal, SIGINT
SIGINT delivered to Python interpreter
Python interpreter reacts by raising KeyboardInterrupt

C signal handler only sets a flag
Nothing more happens until control returns to the main interpreter loop

Main loop checks the flag in some (not all!) bytecode ops
If the flag is set, raises KeyboardInterrupt

¹ Except on Windows

So to reiterate, control-C causes a “signal” to be delivered to CPython. At the time of delivery, the interpreter just sets a flag. Then it waits to get back to a point where it’s actually safe to raise a Python-level exception and unwind the Python call stack.

There’s always a safe point at the beginning of certain bytecode instructions. If you look at bytecodes.c, it’s the ops that include the “check_periodic” component.

As long as we’re only running the code of CPython itself, it never takes very long to get to one of those safe points. But compiled code extensions might introduce large delays, as we’ve seen. Next let’s look at why that is.

What NumPy isn’t doing

def f(rng, out):
    for i in range(len(out)):
        out[i] = rng.random()

 [42]  FOR_ITER        to [96]
       STORE_FAST       1 (i)
       LOAD_FAST        2 (rng)
       LOAD_ATTR        5 (NULL|self + random)
       CALL             0
       LOAD_FAST        0 (out)
       LOAD_FAST        1 (i)
       STORE_SUBSCR
       JUMP_BACKWARD   to [42]
 [96]  END_FOR

voidint
random_standard_uniform_fill(
    bitgen_t *rng,
    npy_intp cnt, double *out)
{
  npy_intp i;
  for (i = 0; i < cnt; i++) {
    out[i] = next_double(rng);
    if (PyErr_CheckSignals())
      return -1;
  }
  return 0;
}

.L3:
    mov    rdi, r12
    call   next_double
    movsd  qword [r13+rbx*8], xmm0
    add    rbx, 1
    cmp    rbp, rbx
    jne    .L3

The delay is because of something NumPy isn’t doing.

On the right here, we have the inner loop that executes when you run numpy.random(n). This is C, so there’s no methods and not really any arrays either, so on the left I show basically equivalent Python. They’re doing the same thing.

Or are they? Here’s the bytecode disassembly for the Python loop. It’s calling the random number generator, and storing the result to index i of the out array, and then doing it again for the next value of i… but see the highlighted bytecode instruction, JUMP_BACKWARD? That’s one of the instructions that checks the flag. So if you hit control-C while this function is running, it will stop after no more than one more iteration.

Now here’s the x86 machine disassembly for the C loop. Same structure, different details, mostly doing the same thing. But the JUMP_BACKWARD equivalent, jne down there at the bottom, it doesn’t check the flag. How could it? Intel and AMD’s hardware engineers don’t design details of CPython into their CPUs!

If we want our compiled code extensions to check that flag, we need to write explicit code to do it.

This is the most basic form of the change we’d need to make to this function.

Why isn’t it doing that?

Abstractly simple
- Call PyErr_CheckSignals periodically
- If it returns −1, treat as failure; abandon work, return

int PyErr_CheckSignals()

Part of the Stable ABI.

This function interacts with Python’s signal handling.

If the function is called from the main thread and under the main Python interpreter, it checks whether a signal has been sent to the processes and if so, invokes the corresponding signal handler. If the signal module is supported, this can invoke a signal handler written in Python.

The function attempts to handle all pending signals, and then returns 0. However, if a Python signal handler raises an exception, the error indicator is set and the function returns −1 immediately (such that other pending signals may not have been handled yet: they will be on the next PyErr_CheckSignals() invocation).

If the function is called from a non-main thread, or under a non-main Python interpreter, it does nothing and returns 0.

This function can be called by long-running C code that wants to be interruptible by user requests (such as by pressing Ctrl-C).

Note: The default Python signal handler for SIGINT raises the KeyboardInterrupt exception.

The first problem I want to highlight is: how are we, extension authors, to know we need to do that?

Here’s the documentation for PyErr_CheckSignals. It’s a big wall of text. I’m gonna highlight the crucial sentence:

“This function can be called by long-running code that wants to be interruptible.” That doesn’t sound nearly as important as it is. I might have written something like “Long-running code needs to call this function frequently in order for the interpreter to be responsive to user interrupts.”

And I’d have made it the first sentence!

Not to mention, before you can read this wall of text and realize its implications, you need to find it.

It’s buried in the “signal handling” section of the exception-handling chapter of the C-API reference manual. The “extending and embedding” guide doesn’t mention it at all.

Why isn’t it doing that?

Abstractly simple
- Call PyErr_CheckSignals periodically
- If it returns −1, treat as failure; abandon work, return
Not obvious from docs that you need to do that

Why isn’t it doing that?

void
random_standard_uniform_fill(
    bitgen_t *rng,
    npy_intp cnt, double *out)
{
  npy_intp i;
  for (i = 0; i < cnt; i++) {
    out[i] = next_double(rng);


  }

}

int
random_standard_uniform_fill(
    bitgen_t *rng,
    npy_intp cnt, double *out)
{
  npy_intp i;
  for (i = 0; i < cnt; i++) {
    out[i] = next_double(rng);
    if (PyErr_CheckSignals())
      return -1;
  }
  return 0;
}

Why isn’t it doing that?

Abstractly simple
- Call PyErr_CheckSignals periodically
- If it returns −1, treat as failure; abandon work, return
Not obvious from docs that you need to do that
Have to decide where and how often to do that
Must propagate the −1 return all the way up your call chain

int PyErr_CheckSignals()

Part of the Stable ABI.

This function interacts with Python’s signal handling.

If the function is called from the main thread and under the main Python interpreter, it checks whether a signal has been sent to the processes and if so, invokes the corresponding signal handler. If the signal module is supported, this can invoke a signal handler written in Python.

The function attempts to handle all pending signals, and then returns 0. However, if a Python signal handler raises an exception, the error indicator is set and the function returns −1 immediately (such that other pending signals may not have been handled yet: they will be on the next PyErr_CheckSignals() invocation).

If the function is called from a non-main thread, or under a non-main Python interpreter, it does nothing and returns 0.

This function can be called by long-running C code that wants to be interruptible by user requests (such as by pressing Ctrl-C).

Note: The default Python signal handler for SIGINT raises the KeyboardInterrupt exception.

Why isn’t it doing that?

int
random_standard_uniform_fill(
    bitgen_t *rng,
    npy_intp cnt, double *out)
{


  npy_intp i;
  for (i = 0; i < cnt; i++) {
    out[i] = next_double(rng);
    if (PyErr_CheckSignals())
      return -1;


  }
  return 0;
}

int
random_standard_uniform_fill(
    bitgen_t *rng,
    npy_intp cnt, double *out)
{
  PyGILState_STATE st;
  int err;
  npy_intp i;
  for (i = 0; i < cnt; i++) {
    out[i] = next_double(rng);
    st = PyGILState_Ensure();
    err = PyErr_CheckSignals();
    PyGILState_Release(st);
    if (err) return err;
  }
  return 0;
}

Why isn’t it doing that?

Abstractly simple
- Call PyErr_CheckSignals periodically
- If it returns −1, treat as failure; abandon work, return
Not obvious from docs that you need to do that
Have to decide where and how often to do that
Must propagate the −1 return all the way up your call chain
PyErr_CheckSignals can run arbitrary Python code
- Needs to be safe to run arbitrary Python
- May need to reclaim the GIL (expensive)

A more complicated example

Cost of checking for signals

What if we didn’t check every time?

Claw back more speed with coarse clock

The third row of this plot shows what we get if we check only once a millisecond. There’s still quite a bit of overhead but not nearly as much as “check whenever possible.”

CLOCK_MONOTONIC is high-precision and that means it has high overhead. Some operating systems offer a “coarse” version that’s plenty good enough to tell us if a millisecond has gone by. That version is efficient enough that the performance cost is barely measurable except for the biggest problem sizes. And notice that “check every 1ms, coarse clock” with the GIL released is no more expensive than “check whenever possible” with the GIL held.

I haven’t tested the new free-threaded interpreter at all yet, by the way. Sorry about that.

Longer intervals don’t help

Recap

CPython interpreter cannot raise KeyboardInterrupt directly from its C-level signal handler
- Must return to main interpreter loop before raising exceptions
- Prompt return needs cooperation from compiled-code extensions
Many extensions do not cooperate
- Need to do this has not been well advertised
- Retrofit requires finicky code changes
- Substantial runtime costs

Short term recommendation

int CheckSignalsOftenEnough(void) {
  static struct timespec last_check = { 0, 0 };
  struct timespec now;
  clock_gettime(CLOCK_MONOTONIC_COARSE, &now);

  if (timespec_difference_at_least(&now, &last_check, ONE_MS_IN_NS)) {
    last_check = now;
    PyGILState_STATE st = PyGILState_Ensure();
    int err = PyErr_CheckSignals();
    PyGILState_Release(st);
    return err;
  }
  return 0;
}

Full version: https://github.com/MillionConcepts/cpython-ext-ctrl-c/blob/main/pycon-2025/CheckSignalsOftenEnough.c

Here’s my short-term recommendation. Take the code from the link at the bottom of the slide and paste it into your extension. You may need to tweak for portability, I’ve only tested it on Linux and FreeBSD with gcc and clang.

Call CheckSignalsOftenEnough at appropriate moments. You still have to figure out where to do that, and how to propagate the failure back out of your extension, but it takes care of looking at the clock and being GIL-safe for you.

It doesn’t take care of any other reason why it might not be safe to run arbitrary Python at the point where you inserted the call, though, like if you’re cutting corners on reference counting.

So. This helps, but can we do better?

I have some ideas, which I will now tell you, in ascending order of zaniness.

How can we make this better?

Improve core documentation
- Highlight need to call PyErr_CheckSignals regularly
- Explain how to do so safely
- Explain how to do so efficiently

How can we make this better?

Don’t require GIL to call PyErr_CheckSignals
- Cost of frequent checks is mostly cost of locking
- Signal flag is already an atomic variable! Don’t need GIL to test it
- Have PyErr_CheckSignals claim GIL itself, only if flag is set
- (Still needs to be safe to take GIL at any callsite)
- Proposed for Python 3.14: gh-133465

How can we make this better?

Make Cython, Numba, and similar tools insert checks for you

cdef random_standard_uniform_fill(
    bitgen_t rng,
    double [:] out,
) nogil:
    for i in range(len(out)):
        out[i] = next_double(rng)

int random_standard_uniform_fill(
    bitgen_t rng,
    __Pyx_memviewslice out
) {
  Py_ssize_t i;
  Py_ssize_t len;
  len = __Pyx_MemoryView_Len(out);
  for (i = 0; i < len; i += 1) {
    *((double *)
      ((out.data +
        i * out.strides[0]))) =
      next_double(rng);
    if (CheckSignalsOftenEnough())
      return -1;
  }
  return 0;
}

How can we make this better?

In tools like PyO3, model control-C as async cancellation

#[pyfunction]
async fn random_standard_uniform_fill(
    #[pyo3(cancel_handle)] mut cancel: CancelHandle,
    rng: BitGen,
    out: &mut f64[],
) {
    futures::select! {
        cancel.cancelled.fuse() => {},
        _ => {
            for i in 0..out.len() {
                out[i] = rng.next_double();
            }
        }
    }
}

How can we make this better?

Turn Python into a shell-structured language

>>> import numpy as np
>>> rng = np.random.default_rng()
>>> def g(n=1000000000):
...     return rng.random(n)
...
>>> g()

Evaluate each step in a subprocess
Pass the result back via shared memory
Let the subprocess die on SIGINT
(This is why the Unix shell doesn’t have the same problem)

If you’re an old Unix beard like me, you may have been thinking, as we went along, “how come the shell doesn’t have this problem?” The sort utility probably doesn’t have any special handling for control-C and yet, when you control-C a pipeline with sort in it, it stops immediately.

The answer is, subprocesses.

Because (we presume) the sort process doesn’t have a handler for SIGINT, instead of “delivering” the signal the kernel just kills off the process, and the shell gets control back immediately.

We could restructure the CPython virtual machine to work the same way. It’d be a huge change … but it would mean that we don’t have to add these checks to every compiled code extension on PyPI!

How can we make this better?

Brainstorming time!
- Call out your ideas
- Or ask a question
- One sentence per person

Acknowledgments

All the code shown in this presentation, the raw data,
and the analysis scripts may be found
at https://github.com/MillionConcepts/cpython-ext-ctrl-c
or https://git.sr.ht/~zackw/cpython-ext-ctrl-c
and may be reused under Million Concepts’ usual 3-clause BSD license
KISS FFT originally written by Mark Borgerding
Data visualization thanks to ggplot2 and cowplot
Graphics postprocessed to various degrees in Inkscape
Slide deck rendered with reveal.js
Fonts are B612 Mono, Libertinus Sans, and Quando
Financial support from NASA ADAP grant 80NSSC21K1421

Writing Extension Modules
to be Interruptible

A Common Bug

What went wrong

Audience background check

What `Control-C` Does¹

Signal Delivery

Signal Delivery

How did we get here???

The black magic of signal delivery

What `Control-C` Does¹

What NumPy isn’t doing

Why isn’t it doing that?

`int PyErr_CheckSignals()`

Why isn’t it doing that?

Why isn’t it doing that?

Why isn’t it doing that?

`int PyErr_CheckSignals()`

Why isn’t it doing that?

Why isn’t it doing that?

A more complicated example

Cost of checking for signals

Cost of checking for signals

What if we didn’t check every time?

What if we didn’t check every time?

Claw back more speed with coarse clock

Longer intervals don’t help

Recap

Short term recommendation

How can we make this better?

How can we make this better?

How can we make this better?

How can we make this better?

How can we make this better?

How can we make this better?

Acknowledgments

Writing Extension Modulesto be Interruptible

A Common Bug

What went wrong

Audience background check

What Control-C Does1

Signal Delivery

Signal Delivery

How did we get here???

The black magic of signal delivery

What Control-C Does1

What NumPy isn’t doing

Why isn’t it doing that?

int PyErr_CheckSignals()

Why isn’t it doing that?

Why isn’t it doing that?

Why isn’t it doing that?

int PyErr_CheckSignals()

Why isn’t it doing that?

Why isn’t it doing that?

A more complicated example

Cost of checking for signals

Cost of checking for signals

What if we didn’t check every time?

What if we didn’t check every time?

Claw back more speed with coarse clock

Longer intervals don’t help

Recap

Short term recommendation

How can we make this better?

How can we make this better?

How can we make this better?

How can we make this better?

How can we make this better?

How can we make this better?

Acknowledgments

Writing Extension Modules
to be Interruptible

What `Control-C` Does¹

What `Control-C` Does¹

`int PyErr_CheckSignals()`

`int PyErr_CheckSignals()`