# Scheduling Guide

## Overview

Symbiont's scheduling system provides production-grade cron-based task execution for AI agents. The system supports:

- **Cron schedules**: Traditional cron syntax for recurring tasks
- **One-shot jobs**: Run once at a specific time
- **Heartbeat pattern**: Continuous assessment-action-sleep cycles for monitoring agents
- **Session isolation**: Ephemeral, shared, or fully isolated agent contexts
- **Delivery routing**: Multiple output channels (Stdout, LogFile, Webhook, Slack, Email, Custom)
- **Policy enforcement**: Security and compliance checks before execution
- **Production hardening**: Jitter, concurrency limits, dead-letter queues, and AgentPin verification

## Architecture

The scheduling system is built on three core components:

```
┌─────────────────────┐
│   CronScheduler     │  Background tick loop (1-second intervals)
│   (Tick Loop)       │  Job selection and execution orchestration
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│   SqliteJobStore    │  Persistent job storage
│   (Job Storage)     │  Transaction support, state management
└──────────┬──────────┘
           │
           ▼
┌─────────────────────┐
│DefaultAgentScheduler│  Agent execution runtime
│ (Execution Engine)  │  AgentContext lifecycle management
└─────────────────────┘
```

### CronScheduler

The `CronScheduler` is the primary entry point. It manages:

- Background tick loop running at 1-second intervals
- Job selection based on next run time
- Concurrency control and jitter injection
- Metrics collection and health monitoring
- Graceful shutdown with in-flight job tracking

### SqliteJobStore

The `SqliteJobStore` provides durable job persistence with:

- ACID transactions for job state updates
- Job lifecycle tracking (Active, Paused, Completed, Failed, DeadLetter)
- Run history with audit trail
- Query capabilities for filtering by status, agent ID, etc.

### DefaultAgentScheduler

The `DefaultAgentScheduler` executes scheduled agents:

- Creates isolated or shared `AgentContext` instances
- Manages session lifecycle (create, execute, destroy)
- Routes delivery to configured channels
- Enforces policy gates before execution

## DSL Syntax

### Schedule Block Structure

Schedule blocks are defined in Symbiont DSL files:

```symbiont
schedule {
  name: "daily-report"
  agent: "reporter-agent"
  cron: "0 0 9 * * *"

  session_mode: "ephemeral_with_summary"
  delivery: ["stdout", "log_file"]

  policy {
    require_approval: false
    max_runtime: "5m"
  }
}
```

### Cron Syntax

Extended cron syntax with six fields (seconds first, optional seventh field for year):

```
┌─────────────── second (0-59)
│ ┌───────────── minute (0-59)
│ │ ┌─────────── hour (0-23)
│ │ │ ┌───────── day of month (1-31)
│ │ │ │ ┌─────── month (1-12)
│ │ │ │ │ ┌───── day of week (0-6, Sunday = 0)
│ │ │ │ │ │
* * * * * *
```

**Examples:**

```symbiont
# Every day at 9 AM
cron: "0 0 9 * * *"

# Every Monday at 6 PM
cron: "0 0 18 * * 1"

# Every 15 minutes
cron: "0 */15 * * * *"

# First day of every month at midnight
cron: "0 0 0 1 * *"
```

### One-Shot Jobs (At Syntax)

For jobs that run once at a specific time:

```symbiont
schedule {
  name: "deployment-check"
  agent: "health-checker"
  at: "2026-02-15T14:30:00Z"  # ISO 8601 timestamp

  delivery: ["webhook"]
  webhook_url: "https://ops.example.com/hooks/deployment"
}
```

### Heartbeat Pattern

For continuous monitoring agents that assess → act → sleep:

```symbiont
schedule {
  name: "system-monitor"
  agent: "heartbeat-agent"
  cron: "0 */5 * * * *"  # Wake every 5 minutes

  heartbeat: {
    enabled: true
    context_mode: "ephemeral_with_summary"
    max_iterations: 100  # Safety limit
  }
}
```

The heartbeat agent follows this cycle:

1. **Assessment**: Evaluate system state (e.g., check metrics, logs)
2. **Action**: Take corrective action if needed (e.g., restart service, alert ops)
3. **Sleep**: Wait until next scheduled tick

## CLI Commands

The `symbi cron` command provides full lifecycle management:

### List Jobs

```bash
# List all jobs
symbi cron list

# Filter by status
symbi cron list --status active
symbi cron list --status paused

# Filter by agent
symbi cron list --agent "reporter-agent"

# JSON output
symbi cron list --format json
```

### Add Job

```bash
# From DSL file
symbi cron add --file agent.symbi --schedule "daily-report"

# Inline definition (JSON)
symbi cron add --json '{
  "name": "quick-task",
  "agent_id": "agent-123",
  "cron_expr": "0 0 * * * *"
}'
```

### Remove Job

```bash
# By job ID
symbi cron remove <job-id>

# By name
symbi cron remove --name "daily-report"

# Force remove (skip confirmation)
symbi cron remove <job-id> --force
```

### Pause/Resume

```bash
# Pause job (stops scheduling, preserves state)
symbi cron pause <job-id>

# Resume paused job
symbi cron resume <job-id>
```

### Status

```bash
# Job details with next run time
symbi cron status <job-id>

# Include last 10 run records
symbi cron status <job-id> --history 10

# Watch mode (auto-refresh every 5s)
symbi cron status <job-id> --watch
```

### Run Now

```bash
# Trigger immediate execution (bypasses schedule)
symbi cron run <job-id>

# With custom input
symbi cron run <job-id> --input "Check production database"
```

### History

```bash
# View run history for a job
symbi cron history <job-id>

# Last 20 runs
symbi cron history <job-id> --limit 20

# Filter by status
symbi cron history <job-id> --status failed

# Export to CSV
symbi cron history <job-id> --format csv > runs.csv
```

## Heartbeat Pattern

### HeartbeatContextMode

Controls how context persists across heartbeat iterations:

```rust
pub enum HeartbeatContextMode {
    /// Fresh context each iteration, append summary to run history
    EphemeralWithSummary,

    /// Shared context across all iterations (memory accumulates)
    SharedPersistent,

    /// Fresh context each iteration, no summary (stateless)
    FullyEphemeral,
}
```

**EphemeralWithSummary (default)**:
- New `AgentContext` per iteration
- Summary of previous iteration appended to context
- Prevents unbounded memory growth
- Maintains continuity for related actions

**SharedPersistent**:
- Single `AgentContext` reused across all iterations
- Full conversation history preserved
- Higher memory usage
- Best for agents needing deep context (e.g., debugging sessions)

**FullyEphemeral**:
- New `AgentContext` per iteration, no carryover
- Lowest memory footprint
- Best for independent checks (e.g., API health probes)

### Heartbeat Agent Example

```symbiont
agent heartbeat_monitor {
  model: "claude-sonnet-4.5"
  system_prompt: """
  You are a system monitoring agent. On each heartbeat:
  1. Check system metrics (CPU, memory, disk)
  2. Review recent error logs
  3. If issues detected, take action:
     - Restart services if safe
     - Alert ops team via Slack
     - Log incident details
  4. Summarize findings
  5. Return 'sleep' when done
  """
}

schedule {
  name: "heartbeat-monitor"
  agent: "heartbeat_monitor"
  cron: "0 */10 * * * *"  # Every 10 minutes

  heartbeat: {
    enabled: true
    context_mode: "ephemeral_with_summary"
    max_iterations: 50
  }

  delivery: ["log_file", "slack"]
  slack_channel: "#ops-alerts"
}
```

## Session Isolation

### Session Modes

```rust
pub enum HeartbeatContextMode {
    /// Ephemeral context with summary carryover (default)
    EphemeralWithSummary,

    /// Shared persistent context across all runs
    SharedPersistent,

    /// Fully ephemeral, no state carryover
    FullyEphemeral,
}
```

**Configuration:**

```symbiont
schedule {
  name: "data-pipeline"
  agent: "etl-agent"
  cron: "0 0 2 * * *"

  # Fresh context per run, summary of previous run included
  session_mode: "ephemeral_with_summary"
}
```

### Session Lifecycle

For each scheduled execution:

1. **Pre-execution**: Check concurrency limits, apply jitter
2. **Session creation**: Create `AgentContext` based on `session_mode`
3. **Policy gate**: Evaluate policy conditions
4. **Execution**: Run agent with input and context
5. **Delivery**: Route output to configured channels
6. **Session cleanup**: Destroy or persist context based on mode
7. **Post-execution**: Update run record, collect metrics

## Delivery Routing

### Supported Channels

```rust
pub enum DeliveryChannel {
    Stdout,           // Print to console
    LogFile,          // Append to job-specific log file
    Webhook,          // HTTP POST to URL
    Slack,            // Slack webhook or API
    Email,            // SMTP email
    Custom(String),   // User-defined channel
}
```

### Configuration Examples

**Single channel:**

```symbiont
schedule {
  name: "backup"
  agent: "backup-agent"
  cron: "0 0 3 * * *"
  delivery: ["log_file"]
}
```

**Multiple channels:**

```symbiont
schedule {
  name: "security-scan"
  agent: "scanner"
  cron: "0 0 1 * * *"

  delivery: ["log_file", "slack", "email"]

  slack_channel: "#security"
  email_recipients: ["ops@example.com", "security@example.com"]
}
```

**Webhook delivery:**

```symbiont
schedule {
  name: "metrics-report"
  agent: "metrics-agent"
  cron: "0 */30 * * * *"

  delivery: ["webhook"]
  webhook_url: "https://metrics.example.com/ingest"
  webhook_headers: {
    "Authorization": "Bearer ${METRICS_API_KEY}"
    "Content-Type": "application/json"
  }
}
```

### DeliveryRouter Trait

Custom delivery channels implement:

```rust
#[async_trait]
pub trait DeliveryRouter: Send + Sync {
    async fn route(
        &self,
        channel: &DeliveryChannel,
        job: &CronJobDefinition,
        run: &JobRunRecord,
        output: &str,
    ) -> Result<(), SchedulerError>;
}
```

## Policy Enforcement

### PolicyGate

The `PolicyGate` evaluates schedule-specific policies before execution:

```rust
pub struct PolicyGate {
    policy_engine: Arc<RealPolicyParser>,
}

impl PolicyGate {
    pub fn evaluate(
        &self,
        job: &CronJobDefinition,
        context: &AgentContext,
    ) -> Result<SchedulePolicyDecision, SchedulerError>;
}
```

### Policy Conditions

```symbiont
schedule {
  name: "production-deploy"
  agent: "deploy-agent"
  cron: "0 0 0 * * 0"  # Sunday midnight

  policy {
    # Require human approval before execution
    require_approval: true

    # Maximum runtime before forced termination
    max_runtime: "30m"

    # Require specific capabilities
    require_capabilities: ["deployment", "production_write"]

    # Time window enforcement (UTC)
    allowed_hours: {
      start: "00:00"
      end: "04:00"
    }

    # Environment restrictions
    allowed_environments: ["staging", "production"]

    # AgentPin verification required
    require_agent_pin: true
  }
}
```

### SchedulePolicyDecision

```rust
pub enum SchedulePolicyDecision {
    Allow,
    Deny { reason: String },
    RequiresApproval { approver: String, reason: String, policy_id: String },
}
```

## Production Hardening

### Jitter

Prevents thundering herd when multiple jobs share a schedule:

```rust
pub struct CronSchedulerConfig {
    pub max_jitter_seconds: u64,  // Random delay 0-N seconds
    // ...
}
```

**Example:**

```toml
[scheduler]
max_jitter_seconds = 30  # Spread job starts across 30-second window
```

### Per-Job Concurrency

Limit concurrent runs per job to prevent resource exhaustion:

```symbiont
schedule {
  name: "data-sync"
  agent: "sync-agent"
  cron: "0 */5 * * * *"

  max_concurrent: 2  # Allow max 2 concurrent runs
}
```

If a job is already running at max concurrency, the scheduler skips the tick.

### Dead-Letter Queue

Jobs exceeding `max_retries` move to `DeadLetter` status for manual review:

```symbiont
schedule {
  name: "flaky-job"
  agent: "unreliable-agent"
  cron: "0 0 * * * *"

  max_retries: 3  # After 3 failures, move to dead-letter
}
```

**Recovery:**

```bash
# List dead-lettered jobs
symbi cron list --status dead_letter

# Review failure reasons
symbi cron history <job-id> --status failed

# Reset job to active after fixing
symbi cron reset <job-id>
```

### AgentPin Verification

Cryptographically verify agent identity before execution:

```symbiont
schedule {
  name: "secure-task"
  agent: "trusted-agent"
  cron: "0 0 * * * *"

  agent_pin_jwt: "${AGENT_PIN_JWT}"  # ES256 JWT from agentpin-cli

  policy {
    require_agent_pin: true
  }
}
```

The scheduler verifies:
1. JWT signature using ES256 (ECDSA P-256)
2. Agent ID matches `iss` claim
3. Domain anchor matches expected origin
4. Expiry (`exp`) is valid

Failures trigger `SecurityEventType::AgentPinVerificationFailed` audit event.

## HTTP API Endpoints

### Schedule Management

**POST /api/v1/schedule**
Create a new scheduled job.

```bash
curl -X POST http://localhost:8080/api/v1/schedule \
  -H "Content-Type: application/json" \
  -d '{
    "name": "hourly-report",
    "agent_id": "reporter",
    "cron_expr": "0 0 * * * *",
    "session_mode": "ephemeral_with_summary",
    "delivery": ["stdout"]
  }'
```

**GET /api/v1/schedule**
List all jobs (filterable by status, agent ID).

```bash
curl "http://localhost:8080/api/v1/schedule?status=active&agent_id=reporter"
```

**GET /api/v1/schedule/{job_id}**
Get job details.

```bash
curl http://localhost:8080/api/v1/schedule/job-123
```

**PUT /api/v1/schedule/{job_id}**
Update job (cron expression, delivery, etc.).

```bash
curl -X PUT http://localhost:8080/api/v1/schedule/job-123 \
  -H "Content-Type: application/json" \
  -d '{"cron_expr": "0 0 */2 * * *"}'
```

**DELETE /api/v1/schedule/{job_id}**
Delete job.

```bash
curl -X DELETE http://localhost:8080/api/v1/schedule/job-123
```

**POST /api/v1/schedule/{job_id}/pause**
Pause job.

```bash
curl -X POST http://localhost:8080/api/v1/schedule/job-123/pause
```

**POST /api/v1/schedule/{job_id}/resume**
Resume paused job.

```bash
curl -X POST http://localhost:8080/api/v1/schedule/job-123/resume
```

**POST /api/v1/schedule/{job_id}/run**
Trigger immediate execution.

```bash
curl -X POST http://localhost:8080/api/v1/schedule/job-123/run \
  -H "Content-Type: application/json" \
  -d '{"input": "Run with custom input"}'
```

**GET /api/v1/schedule/{job_id}/history**
Get run history.

```bash
curl "http://localhost:8080/api/v1/schedule/job-123/history?limit=20&status=failed"
```

**GET /api/v1/schedule/{job_id}/next_run**
Get next scheduled run time.

```bash
curl http://localhost:8080/api/v1/schedule/job-123/next_run
```

### Health Monitoring

**GET /api/v1/health/scheduler**
Scheduler health and metrics.

```bash
curl http://localhost:8080/api/v1/health/scheduler
```

**Response:**

```json
{
  "status": "healthy",
  "active_jobs": 15,
  "paused_jobs": 3,
  "in_flight_jobs": 2,
  "metrics": {
    "runs_total": 1234,
    "runs_succeeded": 1180,
    "runs_failed": 54,
    "avg_execution_time_ms": 850
  }
}
```

## SDK Examples

### JavaScript SDK

```javascript
import { SymbiontClient } from '@symbiont/sdk-js';

const client = new SymbiontClient({
  baseUrl: 'http://localhost:8080',
  apiKey: process.env.SYMBI_API_KEY
});

// Create scheduled job
const job = await client.schedule.create({
  name: 'daily-backup',
  agentId: 'backup-agent',
  cronExpr: '0 0 2 * * *',
  sessionMode: 'ephemeral_with_summary',
  delivery: ['webhook'],
  webhookUrl: 'https://backup.example.com/notify'
});

console.log(`Created job: ${job.id}`);

// List active jobs
const activeJobs = await client.schedule.list({ status: 'active' });
console.log(`Active jobs: ${activeJobs.length}`);

// Get job status
const status = await client.schedule.getStatus(job.id);
console.log(`Next run: ${status.next_run}`);

// Trigger immediate run
await client.schedule.runNow(job.id, { input: 'Backup database' });

// Pause job
await client.schedule.pause(job.id);

// View history
const history = await client.schedule.getHistory(job.id, { limit: 10 });
history.forEach(run => {
  console.log(`Run ${run.id}: ${run.status} (${run.execution_time_ms}ms)`);
});

// Resume job
await client.schedule.resume(job.id);

// Delete job
await client.schedule.delete(job.id);
```

### Python SDK

```python
from symbiont import SymbiontClient

client = SymbiontClient(
    base_url='http://localhost:8080',
    api_key=os.environ['SYMBI_API_KEY']
)

# Create scheduled job
job = client.schedule.create(
    name='hourly-metrics',
    agent_id='metrics-agent',
    cron_expr='0 0 * * * *',
    session_mode='ephemeral_with_summary',
    delivery=['slack', 'log_file'],
    slack_channel='#metrics'
)

print(f"Created job: {job.id}")

# List jobs for specific agent
jobs = client.schedule.list(agent_id='metrics-agent')
print(f"Found {len(jobs)} jobs for metrics-agent")

# Get job details
details = client.schedule.get(job.id)
print(f"Cron: {details.cron_expr}")
print(f"Next run: {details.next_run}")

# Update cron expression
client.schedule.update(job.id, cron_expr='0 */30 * * * *')

# Trigger immediate run
run = client.schedule.run_now(job.id, input='Generate metrics report')
print(f"Run ID: {run.id}")

# Pause during maintenance
client.schedule.pause(job.id)
print("Job paused for maintenance")

# View recent failures
history = client.schedule.get_history(
    job.id,
    status='failed',
    limit=5
)
for run in history:
    print(f"Failed run {run.id}: {run.error_message}")

# Resume after maintenance
client.schedule.resume(job.id)

# Check scheduler health
health = client.schedule.health()
print(f"Scheduler status: {health.status}")
print(f"Active jobs: {health.active_jobs}")
print(f"In-flight jobs: {health.in_flight_jobs}")
```

## Configuration

### CronSchedulerConfig

```rust
pub struct CronSchedulerConfig {
    /// Tick interval (default: 1 second)
    pub tick_interval: Duration,

    /// Global concurrency limit (default: 100)
    pub max_concurrent_cron_jobs: usize,

    /// Persistent job store path (default: None)
    pub job_store_path: Option<PathBuf>,

    /// Catch up missed runs on startup (default: true)
    pub enable_missed_run_catchup: bool,
}
```

### TOML Configuration

```toml
[scheduler]
tick_interval_seconds = 1
max_jitter_seconds = 30
max_concurrent_jobs = 20
enable_metrics = true
default_max_retries = 3
shutdown_timeout_seconds = 60

```

### Delivery channels

Delivery channels (`stdout`, `log_file`, `webhook`, `slack`, `email`, …) are configured **per schedule** in that schedule's `delivery` block (see the examples above), not via a global `[scheduler.delivery]` config section.

> **`log_file` delivery requires `SYMBIONT_LOG_DIR`.** The destination path is confined to that allowlisted directory — paths that escape it via `..`, absolute paths, or symlinks are rejected, and log-file delivery is fail-closed (disabled) when `SYMBIONT_LOG_DIR` is unset.

## Observability

### Metrics (Prometheus-compatible)

```
# Total runs
symbiont_cron_runs_total{job_name="daily-report",status="succeeded"} 450

# Failed runs
symbiont_cron_runs_total{job_name="daily-report",status="failed"} 5

# Execution time histogram
symbiont_cron_execution_duration_seconds{job_name="daily-report"} 1.234

# In-flight jobs gauge
symbiont_cron_in_flight_jobs 3

# Dead-lettered jobs
symbiont_cron_dead_letter_total{job_name="flaky-job"} 2
```

### Audit Events

All scheduler actions emit security events:

```rust
pub enum SecurityEventType {
    CronJobCreated,
    CronJobUpdated,
    CronJobDeleted,
    CronJobPaused,
    CronJobResumed,
    CronJobExecuted,
    CronJobFailed,
    CronJobDeadLettered,
    AgentPinVerificationFailed,
}
```

Query the audit log from the interactive shell:

```text
/audit CronJobFailed
```

Or programmatically via the runtime HTTP API — see [API Reference](/api-reference) for the `/api/v1/audit` endpoints.

## Best Practices

1. **Use jitter for shared schedules**: Prevent multiple jobs from starting simultaneously
2. **Set concurrency limits**: Protect against resource exhaustion
3. **Monitor dead-letter queue**: Review and fix failing jobs regularly
4. **Use EphemeralWithSummary**: Prevents unbounded memory growth in long-running heartbeats
5. **Enable AgentPin verification**: Cryptographically verify agent identity
6. **Configure delivery routing**: Use appropriate channels for different job types
7. **Set policy gates**: Enforce time windows, approvals, and capability checks
8. **Use heartbeat pattern for monitoring**: Continuous assessment-action-sleep cycles
9. **Test schedules in staging**: Validate cron expressions and job logic before production
10. **Export metrics**: Integrate with Prometheus/Grafana for operational visibility

## Troubleshooting

### Job Not Running

1. Check job status: `symbi cron status <job-id>`
2. Verify cron expression: Use [crontab.guru](https://crontab.guru/)
3. Check scheduler health: `curl http://localhost:8080/api/v1/health/scheduler`
4. Review logs: `symbi logs --filter scheduler --level debug`

### Job Failing Repeatedly

1. View history: `symbi cron history <job-id> --status failed`
2. Check error messages in run records
3. Verify agent configuration and capabilities
4. Test agent outside scheduler: `symbi run <agent-id> --input "test"`
5. Check policy gates: Ensure time windows and capabilities match

### Dead-Lettered Job

1. List dead-letter jobs: `symbi cron list --status dead_letter`
2. Review failure pattern: `symbi cron history <job-id>`
3. Fix root cause (agent code, permissions, external dependencies)
4. Reset job: `symbi cron reset <job-id>`

### High Memory Usage

1. Check session mode: Switch to `ephemeral_with_summary` or `fully_ephemeral`
2. Reduce heartbeat iterations: Lower `max_iterations`
3. Monitor context size: Review agent output verbosity
4. Enable context archiving: Configure retention policies

## Migration from v0.9.0

The v1.0.0 release adds production hardening features. Update your job definitions:

```diff
 schedule {
   name: "my-job"
   agent: "my-agent"
   cron: "0 0 * * * *"
+
+  # Add concurrency limit
+  max_concurrent: 2
+
+  # Add AgentPin for identity verification
+  agent_pin_jwt: "${AGENT_PIN_JWT}"
+
+  policy {
+    require_agent_pin: true
+  }
 }
```

Update configuration:

```diff
 [scheduler]
 tick_interval_seconds = 1
+ max_jitter_seconds = 30
+ default_max_retries = 3
+ shutdown_timeout_seconds = 60
```

No breaking API changes. All v0.9.0 jobs continue to work.
