---
id: RL-10
title: "Graceful shutdown (SIGINT / SIGTERM drain, checkpoint, close)"
status: Pending
type: implementation
blocked_by: [RL-2, RL-8, RL-9]
unlocks: []
severity: High
target_repo: plexus-substrate
---

## Problem

Substrate has no graceful shutdown. On SIGINT (Ctrl-C) or SIGTERM the tokio runtime drops, in-flight Orcha graphs terminate abruptly with no checkpoint, SQLite pools close implicitly (no explicit `close().await`), spawned tasks are force-aborted by the runtime with no cleanup path, and there is no observable "shutting down" state for clients to respect.

Audit summary: "No graceful shutdown. No activation implements `Drop` or `close()` / `shutdown()`. SQLite pools close implicitly. In-flight Orcha graphs terminate abruptly on SIGINT."

On SIGINT / SIGTERM, substrate should:

1. Stop accepting new requests.
2. Cancel in-flight requests using the mechanism landed in RL-9.
3. Let each activation's shutdown hook run to completion (or hit a bounded timeout).
4. Explicitly close SQLite pools (sqlx's `Pool::close().await`).
5. Drain the plugin-system task set and Health's background task handles established in RL-8.
6. Exit with status 0 within a bounded window (pin at 5 s unless a per-activation shutdown documents a longer budget).

## Context

**Depends on RL-2** (typed startup errors, so the binary entry point can own a `Result`-returning main and a signal handler without re-introducing `.expect`).

**Depends on RL-8** (every task has an owner, so shutdown has something to await / abort).

**Depends on RL-9** (the unified `CancellationToken` mechanism; RL-10 signals the top-level token and every activation's token is its child).

**Signal handling.** Use `tokio::signal::ctrl_c()` for SIGINT and `tokio::signal::unix::signal(SignalKind::terminate())` for SIGTERM (Linux / macOS only — Windows handling is out of scope; substrate targets macOS per the audit note).

**Activation shutdown contract.** Each activation with in-flight work or open resources exposes `async fn shutdown(&self) -> Result<(), ActivationError>`. The builder holds the list of activations in registration order; on shutdown, iterate in *reverse* order (so children shut down before parents, if any parent/child relationships exist at the hub level). Each activation's shutdown has its own bounded window (default 2 s; implementor can override per activation with a documented reason).

**Orcha checkpoint.** Orcha's shutdown hook persists the current graph state using its existing `pm.save_*` machinery (now RL-5-hardened) so in-flight graphs resume cleanly on next startup. If a save fails during shutdown, log at ERROR and continue — do not block shutdown on a persistence failure.

**SQLite close.** `sqlx::Pool::close().await` is async and idempotent. Each storage owner calls it in its shutdown hook.

## Required behavior

| Event | Current observable behavior | Required observable behavior |
|---|---|---|
| SIGINT received with no in-flight work | Runtime drops; process exits | Process exits with status 0 within 500 ms. |
| SIGINT received with an in-flight Orcha graph mid-step | Graph state abandoned mid-step; no checkpoint | Graph's current state is persisted via `pm.save_*`; process exits with status 0 within 5 s. |
| SIGTERM received with streaming calls active | Streaming tasks force-aborted by runtime | Streaming activations observe cancellation via RL-9's token, clean up, exit; process exits with status 0 within 5 s. |
| SIGINT received and an activation's shutdown exceeds its window | Process hangs (today: not applicable; there is no shutdown path) | Shutdown proceeds to the next activation; offending activation is logged at ERROR naming it; process still exits within the overall 5 s budget. |
| Shutdown completes | SQLite pools close implicitly | Every SQLite pool is explicitly closed via `Pool::close().await`. |
| New request arrives during shutdown | Normal handling (no concept of "draining") | Request is rejected with a structured "server shutting down" error; transport returns this to the caller. |

## Risks

- **Shutdown ordering deadlock.** If activation A's shutdown depends on activation B being alive (e.g., Orcha's checkpoint calls Loopback), but B is shut down first, the save fails. Mitigation: iterate *in reverse registration order*, which matches the dependency direction already enforced by cyclic-parent injection. If a latent cycle exists, it surfaces here as a shutdown failure that should be fixed in DC epic — not this ticket.
- **SIGTERM delivery on macOS test.** Some CI environments don't reliably deliver SIGTERM to a child `cargo run`. The acceptance test uses SIGINT (trivially deliverable via Ctrl-C equivalent) for the primary check; SIGTERM is exercised via a separate test if a Unix-process-spawn harness is available.
- **Shutdown during startup.** If SIGINT arrives before all activations have initialised, the partial-init activations may not have a shutdown hook yet. Handle by checking each activation's state before calling its shutdown; log and skip if not yet initialised.

## What must NOT change

- The set of RPC method names or request/response shapes on any activation.
- Startup order in `src/builder.rs`.
- SQLite-per-activation layout or schema.
- Normal operation when no signal is received (regression).
- Existing `cargo test` pass rate.

## Acceptance criteria

1. `src/main.rs` (or equivalent binary entry point) registers a SIGINT handler and a SIGTERM handler (Unix only).
2. Every activation with spawned tasks or open SQLite pools implements `async fn shutdown(&self) -> Result<(), ActivationError>` (name may vary per the error-enum conventions set in RL-3 / RL-5); the builder wires these in reverse registration order.
3. A manual integration test: start substrate, begin an Orcha graph that will take > 2 s to complete, send SIGINT, observe: process exits with status 0 within 5 s, stderr contains a "shutting down" log line, next startup resumes the graph from the checkpointed state.
4. A unit test spawns a test substrate, triggers the shutdown path directly (without a real signal), and asserts every activation's shutdown hook was called and every SQLite pool's `is_closed()` returns true.
5. A new-request-during-shutdown test: trigger shutdown, then attempt a call against any activation; assert the call receives a structured "server shutting down" error (the exact error variant is pinned in this ticket's implementation PR).
6. All existing `cargo test` targets pass.
7. `plans/README.md`'s roadmap mentions RL-10 as the final RL phase (optional update; acceptable if `RL-1.md`'s Completion section is the authoritative record).

## Completion

Implementor delivers:

- Patch to `src/main.rs` adding signal handling and driving shutdown.
- Patch to `src/builder.rs` adding the reverse-order shutdown loop and "draining" state tracking.
- Per-activation `shutdown` implementations (Orcha with checkpoint, every SQLite-owning activation with explicit pool close, Health / plugin system with task-set drain from RL-8).
- A new structured error variant (e.g., `SubstrateError::ShuttingDown` or per-activation equivalent) for new requests arriving during drain.
- Manual integration test per criterion 3 (test script committed under `tests/` or documented in the PR body).
- Unit test per criteria 4 and 5.
- `cargo test` output confirming criterion 6.
- Status flip to `Complete` in the same commit that lands the code.
