Raic

Raic targets ~90% fewer tokens than idiomatic Rust for the same program, with bi-directional, deterministic compilation: T-AST (Tokenized AST) ↔ Rust.

Raic is not a traditional human-readable scripting syntax. It is an AST serialization format tuned for BPE (Byte-Pair Encoding) tokenizers: a postfix (stack-based) combinator grammar plus a large, single-token opcode dictionary. Rust’s expression tree is represented explicitly—syntactic sugar, arbitrary variable names, and brace nesting are stripped in favor of stack operations and macro-level opcodes that expand to fixed, idiomatic Rust patterns.

• T-AST → Rust: deterministic expansion of signatures and opcode sequences into idiomatic Rust (same structure every time for a given opcode).
• Rust → T-AST: parse the Rust AST and collapse recognized idioms into single opcodes (e.g. a frequency-counting for loop → Fm, a tuple second-field descending sort → Sd).

LLMs are strong at order-sensitive, context-conditioned sequences; they can learn to emit balanced stack programs without needing {, }, or named locals in the surface form.

1. Type dictionary (single-token types)

These align with short, tokenizer-friendly spellings (often one BPE token each in code-heavy models):

Full Rust type names remain valid wherever the pipeline accepts types.

2. Declarations (defaults over ceremony)

Visibility: everything is pub by default; private items use the existing _-prefix convention where applicable.

Function — name, argument types (stack inputs), return type (stack output):

name arg1_type arg2_type : return_type = opcode_body

Example:

top_words s N : [(S N)] = Ws Fm Vi Sd Tk

Rust equivalent (conceptually):

pub fn top_words(arg0: &str, arg1: usize) -> Vec<(String, usize)>

(argument names are synthesized deterministically, e.g. text / n, when emitting readable Rust).

Struct:

@ Name field1:T field2:T

Example:

@ User id:N name:S

Rust equivalent:

pub struct User { pub id: usize, pub name: String }

3. Thick AST: opcode vocabulary

Instead of spelling for, let, +, and closures in full text, T-AST uses a large dictionary of single-token opcodes (symbols or short words) that map to complex, idiomatic Rust AST fragments.

Illustrative opcodes (the real dictionary grows with the compiler):

Control / combinators (stack discipline):

Exact tokenization of [ ] blocks and postfix bodies is specified in the compiler and tests/integration.rs.

4. Bi-directional example: top_words

Hand-written Rust (illustrative; token count depends on BPE):

pub fn top_words(text: &str, n: usize) -> Vec<(String, usize)> {
    let mut counts: HashMap<String, usize> = HashMap::new();
    for word in text.split_whitespace() {
        *counts.entry(word.to_lowercase()).or_insert(0) += 1;
    }
    let mut pairs: Vec<_> = counts.into_iter().collect();
    pairs.sort_by(|a, b| b.1.cmp(&a.1));
    pairs.into_iter().take(n).collect()
}

T-AST (10 tokens in the canonical spelling shown in tests):

top_words s N : [(S N)] = Ws Fm Vi Sd Tk

How it composes:

• Rust → T-AST: Match the for frequency loop → Fm; match the sort_by on tuple .1 reversed → Sd; strip local names; order arguments as s then N on the stack.
• T-AST → Rust: Emit text / n (or arg0 / arg1 per emitter rules). Ws consumes the string slice; Fm emits the map loop; Vi collects; Sd emits the sort closure; Tk wires take + collect with n.

No semantic hiding: Fm is not a vague shortcut—it is the agreed, canonical expansion for “count into HashMap with this loop shape.” Scaffolding is what gets removed.

Determinism: Each opcode expands to one idiomatic Rust pattern from the dictionary—no ambiguous sugar.

Compression: Removing names, closures-as-text, and braces removes BPE-heavy surface; what remains is the semantic skeleton.

Examples (T-AST cookbook)

The reference compiler accepts one-line struct declarations, function lines with name args : ret = opcodes, and a growing opcode dictionary. Items are separated by blank lines where needed.

Structs

@ User id:N name:S

@ Post slug:S title:S body:S

@ Config site_title:S base_url:S

Functions (supported opcode chain)

The word-frequency / top-n pipeline (see integration tests):

top_words s N : [(S N)] = Ws Fm Vi Sd Tk

Same chain, different name:

rank_terms s N : [(S N)] = Ws Fm Vi Sd Tk

Multi-item file

@ Site name:S url:S

top_words s N : [(S N)] = Ws Fm Vi Sd Tk

Types you can compose

Compiling

raic compile main.rc -o generated.rs

Use main.rc (or any path) for T-AST source; pass -o for the Rust output file.

CLI reference (quick)

5. Implementation: pattern matching vs. rule files

The Rust ↔ T-AST pipeline needs AST pattern matching to recognize and collapse idioms into opcodes, and deterministic expansion on the way back.

That logic can be:

1. Hardcoded in the compiler (fast to ship, easy to test opcode-by-opcode), or
2. Data-driven from a textual dictionary of rewrite rules (easier to extend without recompiling, more moving parts).

The repo can start with (1) and migrate hot spots to (2) once the opcode set stabilizes.

Getting started

Build the compiler:

cargo build --release

Or install it:

cargo install --path .

Compile T-AST to Rust:

raic compile main.rc -o generated.rs

The full CLI:

Omit FILE or pass - to read from stdin.

Syntax highlighting

TextMate grammar: syntaxes/raic.tmLanguage.json · VS Code: editors/vscode-raic.

HTML docs:

npm ci
npm run docs:html

Or make docs-html.

Teaching an LLM

Use PROMPT.md for narrative context. The executable T-AST contract is tests/integration.rs. For multi-model evaluations (same task in Raic vs Rust), see llm_eval/README.md and run uv sync --extra llm-eval then python llm_eval/run_eval.py.

Measuring

make token-stats

(requires uv) — compares token counts for sources vs. emitted Rust.

Status

v0.3 — pre-1.0. The language is T-AST as above; the compiler and tests are the source of truth. Pin versions for stability across revisions.