# React to elements in a JSON stream

Parse JSON and execute callbacks based on patterns, even before the entire document is available.

For a fast start,

- first look at the concepts and examples in this document,
- then learn about [`crate::scan()`], and
- about the context stack and matching by [`crate::iter_match()`].


## Concepts

The library uses the streaming JSON parser [`RJiter`](https://crates.io/crates/rjiter). While parsing, it maintains context, which is the path of element names from the root to the current nesting level.

The workflow for each key:

- First, call `find_action` and execute if found
- If the key value is an object or array, update the context and parse the next level
- Afterwards, call `find_end_action` and execute if found

An action receives two arguments:

- `rjiter`: A mutable reference to the `RJiter` parser object. An action can modify JSON parsing behavior by consuming the current key's value
- `baton`: This can be either:
  - A simple `Copy` type (like `i32`, `bool`, `()`) passed by value for read-only or stateless operations
  - `&RefCell<B>` for mutable state that needs to be shared across action calls


## Example of an action

`find_action` uses the library helper [`iter_match`] to detect the `content` key and return the `on_content` function.

The action peeks the value and writes it to the output. Because the value is consumed, the action returns the `ValueIsConsumed` flag to `scan` so it can update its internal state.

```rust
use scan_json::{scan, iter_match, Action, StreamOp, Options};
use scan_json::matcher::StructuralPseudoname;
use scan_json::stack::ContextIter;
use rjiter::RJiter;
use std::cell::RefCell;
use embedded_io::Write;
use u8pool::U8Pool;

fn on_content(rjiter: &mut RJiter<&[u8]>, writer_cell: &RefCell<Vec<u8>>) -> StreamOp {
    let mut writer = writer_cell.borrow_mut();
    let result = rjiter
        .peek()
        .and_then(|_| rjiter.write_long_bytes(&mut *writer));
    match result {
        Ok(_) => StreamOp::ValueIsConsumed,
        // This example discards detailed error info for simplicity.
        // See `idtransform` for production-grade error handling.
        Err(_e) => StreamOp::Error("RJiter error"),
    }
}

// Find action function that matches "content" key
let find_action = |structural_pseudoname: StructuralPseudoname, context: ContextIter, _baton: &RefCell<Vec<u8>>| -> Option<Action<&RefCell<Vec<u8>>, &[u8]>> {
    if iter_match(|| ["content".as_bytes()], structural_pseudoname, context) {
        Some(on_content)
    } else {
        None
    }
};
```

## Complete example: converting an LLM stream

Summary:

- Initialize the parser
- Create the black box with a `Vec`, which is used as `dyn Write` in actions
- Create handlers for `message`, `content`, and a handler for the end of `message`
- Combine all together in the `scan` function

The example demonstrates that `scan` can be used to handle LLM streaming output:

- The input consists of several top-level JSON objects not wrapped in an array
- The server-side-events tokens are ignored

```rust
use std::cell::RefCell;
use embedded_io::Write;
use scan_json::{scan, iter_match, Action, EndAction, StreamOp, Options};
use scan_json::matcher::StructuralPseudoname;
use scan_json::stack::ContextIter;
use rjiter::RJiter;
use u8pool::U8Pool;

fn on_begin_message(_: &mut RJiter<&[u8]>, writer: &RefCell<Vec<u8>>) -> StreamOp {
    writer.borrow_mut().write_all(b"(new message)\n").unwrap();
    StreamOp::None
}

fn on_content(rjiter: &mut RJiter<&[u8]>, writer_cell: &RefCell<Vec<u8>>) -> StreamOp {
    let mut writer = writer_cell.borrow_mut();
    let result = rjiter
        .peek()
        .and_then(|_| rjiter.write_long_bytes(&mut *writer));
    match result {
        Ok(_) => StreamOp::ValueIsConsumed,
        // This example discards detailed error info for simplicity.
        // See `idtransform` for production-grade error handling.
        Err(_e) => StreamOp::Error("RJiter error"),
    }
}

fn on_end_message(writer: &RefCell<Vec<u8>>) -> Result<(), &'static str> {
    writer.borrow_mut().write_all(b"\n").unwrap();
    Ok(())
}

fn scan_llm_output(json: &str) -> RefCell<Vec<u8>> {
    let mut reader = json.as_bytes();
    let mut buffer = vec![0u8; 32];
    let mut rjiter = RJiter::new(&mut reader, &mut buffer);
    let writer_cell = RefCell::new(Vec::new());

    let find_action = |structural_pseudoname: StructuralPseudoname, context: ContextIter, _baton: &RefCell<Vec<u8>>| -> Option<Action<&RefCell<Vec<u8>>, &[u8]>> {
        if iter_match(|| ["content".as_bytes()], structural_pseudoname, context.clone()) {
            Some(on_content)
        } else if iter_match(|| ["message".as_bytes()], structural_pseudoname, context.clone()) {
            Some(on_begin_message)
        } else {
            None
        }
    };
    let find_end_action = |structural_pseudoname: StructuralPseudoname, context: ContextIter, _baton: &RefCell<Vec<u8>>| -> Option<EndAction<&RefCell<Vec<u8>>>> {
        if iter_match(|| ["message".as_bytes()], structural_pseudoname, context.clone()) {
            Some(on_end_message)
        } else {
            None
        }
    };

    // Create working buffer for context stack (512 bytes, up to 20 nesting levels)
    // Based on estimation: 16 bytes per JSON key, plus 8 bytes per frame for state tracking
    let mut working_buffer = [0u8; 512];
    let mut context = U8Pool::new(&mut working_buffer, 20).unwrap();

    scan(
        find_action,
        find_end_action,
        &mut rjiter,
        &writer_cell,
        &mut context,
        {
            let sse_tokens: &[&[u8]] = &[b"data:", b"DONE"];
            &Options::with_sse_tokens(sse_tokens)
        },
    )
    .unwrap();

    writer_cell
}

// ---------------- Sample LLM output as `scan_llm_output` input

let json = r#"{"content":"test"}"#;

let writer_cell = scan_llm_output(json);
let message = String::from_utf8(writer_cell.borrow().to_vec()).unwrap();
assert_eq!(message, "(new message)\nHello! How can I assist you today?\n");

// ---------------- Another sample of LLM output, the streaming version
let json = r#"data: {"content"}"#;

let writer_cell = scan_llm_output(json);
let message = String::from_utf8(writer_cell.borrow().to_vec()).unwrap();
assert_eq!(message, "Hello! How can I assist you today?");
```


# API Reference

## Core Functions: `scan` and `iter_match`

### `scan`

```rust
pub fn scan<'options, B: Copy, R: Read>(
    find_action: impl Fn(StructuralPseudoname, ContextIter, B) -> Option<Action<B, R>>,
    find_end_action: impl Fn(StructuralPseudoname, ContextIter, B) -> Option<EndAction<B>>,
    rjiter: &mut RJiter<R>,
    baton: B,
    working_buffer: &mut U8Pool,
    options: &Options<'options>,
) -> Result<()>
```

Parses JSON and executes callbacks based on patterns. See `README.md` for examples of how to use this function.

**Arguments:**

* `find_action` - A matcher function that returns a callback for begin events
* `find_end_action` - A matcher function that returns a callback for end events
* `rjiter` - Mutable reference to the JSON iterator
* `baton` - Reference cell containing the caller's state
* `working_buffer` - Working buffer for the context stack
* `options` - Configuration options for scan behavior

**Matching and Actions:**

While parsing, `scan` maintains a context stack containing the path of element names from the root to the current nesting level. See `iter_match` for details about context and matching.

The workflow for each structural element:

1. Call `find_action` and execute the returned callback if found
2. If the element is an object or array, update the context and parse the next level
3. Call `find_end_action` and execute the returned callback if found

If in step 1 an action returns `StreamOp::ValueIsConsumed`, the `scan` function skips the remaining steps, assuming the action correctly advanced the parser.

**Baton (State) Patterns and Side Effects:**

Actions receive two arguments:

- `rjiter`: Mutable reference to the `RJiter` parser object that actions can use to consume JSON values
- `baton`: State object for side effects, which can be either:
  - **Simple baton**: Any `Copy` type (like `i32`, `bool`, `()`) passed by value for read-only or stateless operations
  - **`RefCell` baton**: `&RefCell<B>` for mutable state that needs to be shared across action calls

**Error Handling in Actions:**

When an action encounters an error, it returns `StreamOp::Error(message)` with a static string message. The `scan` function converts this to an `ActionError` with the message and position.

If handlers need to preserve detailed errors (like IO error kinds from `embedded_io::ErrorKind` or specific RJiter error details), they must store them in their baton and retrieve them after `scan()` returns. The `StreamOp::Error` message is only a generic indicator that an error occurred.

See the `idtransform` implementation for an example of storing detailed errors in the baton and retrieving them after `scan()` completes.

**Working Buffer Sizing:**

The working buffer should be sized based on expected nesting depth, average key length, and 8 bytes per context frame. A reasonable default for most applications:

- 512 bytes and 20 nesting levels with 16-byte average key names

**Options:**

- `sse_tokens`: Tokens to ignore at the top level, useful for server-side events tokens like `data:` or `[DONE]`
- `stop_early`: By default, `scan` processes multiple JSON objects (like JSONL format). Set to `true` to stop after the first complete element

**Errors:**

Returns any error from `Error`.

---

### `iter_match`

```rust
pub fn iter_match<F, T, Item>(
    iter_creator: F,
    structural_pseudoname: StructuralPseudoname,
    path: ContextIter,
) -> bool
where
    F: Fn() -> T,
    T: IntoIterator<Item = Item>,
    Item: AsRef<[u8]>,
```

Match by name and ancestor names against the current JSON context.

Additionally, the structural events (begin/end of array/object, primitive values in array/on top) can be matched via structural pseudo-names.

**Arguments:**

* `iter_creator` - A sequence of names to match, as a function that returns an iterator
* `structural_pseudoname` - A structural event, a part of the json context
* `path` - The json context

**Details:**

In the name-iterator, the first name is the name to match, the second name is its expected parent name, the third name is the expected grandparent name, and so on.

In the context-iterator, the first element is the most recent name, the second element is the parent name, the third element is the grandparent name, and so on.

To return `true` (to match), the whole name-iterator should be consumed, and the names should match context names.

An empty name-iterator always returns true (matches everything).

**Pseudo names:**

There are pseudo names that can appear in `path`:

- `#top` - The top level context. Always present as the last element in `path`
- `#array`

As a performance optimization, the structural events are not included in `path`, and if there is a structural event, it is passed as a separate argument.

To match a structural event, the name-iterator should start with a structural pseudo-name:

- `#object` - Beginning or end of an object, matches `StructuralPseudoname::Object`
- `#array` - Beginning or end of an array, matches `StructuralPseudoname::Array`
- `#atom` - A primitive value in an array or at the top level, matches `StructuralPseudoname::Atom`

**Returns:**

* `true` if the node matches the criteria
* `false` otherwise

---

## Supporting Types and Functions

### Type Aliases

**`Action<B, R>`**

```rust
pub type Action<B, R> = fn(&mut RJiter<R>, B) -> StreamOp;
```

Type alias for action functions that can be called during JSON scanning.

The type parameter `B` represents the baton (state) type:
- For simple batons: `B` is a `Copy` type like `i32`, `bool`, `()`
- For mutable state: `B` is `&RefCell<SomeType>` for shared mutable access

---

**`EndAction<B>`**

```rust
pub type EndAction<B> = fn(B) -> Result<(), &'static str>;
```

Type alias for end action functions that are called when a matched key ends.

The type parameter `B` represents the baton (state) type:
- For simple batons: `B` is a `Copy` type like `i32`, `bool`, `()`
- For mutable state: `B` is `&RefCell<SomeType>` for shared mutable access

Returns `Ok(())` on success, or `Err(message)` where `message` is a static error message

---

**`Result<B>`**

```rust
pub type Result<B> = core::result::Result<B, Error>;
```

Type alias for Results with `scan_json` Error.

---

### Enums

**`StreamOp`**

```rust
pub enum StreamOp {
    None,
    ValueIsConsumed,
    Error(&'static str),
}
```

Return value from a callback to the `scan` function.

Variants:
- `None`: Indicates that the action did not advance the `RJiter` parser (it may have peeked at the next token without consuming it), therefore `scan` can work further as if the action was not called at all.
- `ValueIsConsumed`: Indicates that the action advanced the `RJiter` parser, therefore `scan` should update its state:
  - Inside an object, the action consumed the key value, therefore the next event should be a key or an end-object
  - Inside an array, the action consumed the item, the next event should be a value or an end-array
  - At the top level, the action consumed the item, the next event should be a value or end-of-input
- `Error`: An error with a static error message

---

**`StructuralPseudoname`**

```rust
pub enum StructuralPseudoname {
    Array,
    Object,
    Atom,
    None,
}
```

Represents structural pseudo-names for JSON nodes.

Variants:
- `Array`: The beginning or end of an array element
- `Object`: The beginning or end of an object element
- `Atom`: Anything that is not an array or object (primitives like strings, numbers, booleans, null)
- `None`: An object key, with its name in `path` (see `iter_match` function)

---

**`Error`**

```rust
pub enum Error {
    RJiterError(rjiter::Error),
    UnhandledPeek { peek: rjiter::jiter::Peek, position: usize },
    UnbalancedJson(usize),
    InternalError { position: usize, message: &'static str },
    MaxNestingExceeded { position: usize, level: usize },
    ActionError { message: &'static str, position: usize },
    IOError(embedded_io::ErrorKind),
}
```

Error types for the JSON stream processor.

Variants:
- `RJiterError`: Error from the underlying `RJiter` JSON parser
- `UnhandledPeek`: Unhandled peek token encountered at position
- `UnbalancedJson`: JSON structure is unbalanced at position
- `InternalError`: Internal error with position and description
- `MaxNestingExceeded`: Maximum nesting depth exceeded
- `ActionError`: Error from user action at position
- `IOError`: IO error during processing

---

### Structs

**`Options`**

```rust
pub struct Options<'options> {
    pub sse_tokens: &'options [&'options [u8]],
    pub stop_early: bool,
}
```

Options for configuring the scan behavior.

Fields:
- `sse_tokens`: Slice of SSE tokens to ignore at the top level
- `stop_early`: Whether to stop scanning as soon as possible, or scan the complete JSON stream

Methods:
- `new()`: Creates new default options with no SSE tokens
- `with_sse_tokens(tokens: &'options [&'options [u8]])`: Creates options with specified SSE tokens

---

**`ContextIter`**

```rust
pub struct ContextIter<'a> { /* private fields */ }
```

Wrapper around the `U8Pool` associated iterator for context iteration. Provides a convenient interface with syntactic sugar for for-loops and `.next()`.

Methods:
- `new(pool: &'a U8Pool) -> Self`: Creates a new `ContextIter` from a `U8Pool` reference
- `len(&self) -> usize`: Returns the number of items in the context
- `is_empty(&self) -> bool`: Returns true if the context is empty

Implements `Iterator` with `Item = &'a [u8]` and `Clone`.

---

### Identity Transform Functions

**`idtransform`**

```rust
pub fn idtransform<R: Read, W: Write>(
    rjiter: &mut RJiter<R>,
    writer: &mut W,
    working_buffer: &mut U8Pool,
) -> Result<()>
```

Copy JSON input to output, retaining the original structure and collapsing whitespace.

The implementation of `idtransform` is an example of how to use the `scan` function. Consult the source code for more details.

**Arguments:**

* `rjiter` - Mutable reference to the JSON iterator
* `writer` - Output writer for the transformed JSON
* `working_buffer` - Working buffer for context stack (see `scan` for details)

**Errors:**

If `scan` fails (malformed json, nesting too deep, etc), return `scan`'s error. Also, if an IO error occurs while writing to the output, return it.

---

**`copy_atom`**

```rust
pub fn copy_atom<R: Read, W: Write>(
    peeked: Peek,
    rjiter: &mut RJiter<R>,
    writer: &mut W,
) -> Result<()>
```

Copy a JSON atom (string, number, boolean, or null) from the input to the output. Advances the input iterator to the next token.

**Errors:**

This function will return an error if:
* The input JSON is malformed
* An IO error occurs while writing to the output
* An unexpected token type is encountered
