AWKRSALL(1)

NAME
awkrsall - the awkrs comprehensive reference (invocation, language, bytecode VM, Cranelift JIT, persistent bytecode cache, parallel records, gawk/mawk parity)

OVERVIEW
This is the all-in-one awkrs manual, modeled on zshall(1). It concatenates the major sections of the awkrs reference into a single page so you can grep with `/`:

    1. INVOCATION — POSIX, GNU, gawk, mawk -W flags
    2. PROGRAM TEXT — rules, patterns, statements, getline, operators
    3. DATA MODEL — fields, arrays, records, FS/FPAT/RS, special vars
    4. BUILTINS — every string/numeric/I/O/bit builtin
    5. GAWK EXTENSIONS — @include, @load, @namespace, /inet/, indirect calls
    6. BYTECODE VM — stack VM, opcodes, peephole fusion, inline fast paths
    7. CRANELIFT JIT — tiered compilation, ABI, eligible ops, NaN-boxing
    8. BYTECODE CACHE — rkyv, mmap, atomic rename, mtime invalidation
    9. PARALLEL RECORDS — static safety check, rayon, deterministic reorder
    10. BENCHMARKS — vs gawk / mawk / BSD awk / frawk / goawk
    11. ENVIRONMENT
    12. EXIT STATUS
    13. FILES
    14. EXAMPLES — throughput, CSV, parallel, JIT-friendly, regex
    15. PARITY GAPS

For a shorter quick-reference page see awkrs(1).

NAME
awkrs - pattern-directed scanning and processing language with bytecode VM, Cranelift JIT, persistent rkyv bytecode cache, and parallel record processing (Rust awk implementation)

SYNOPSIS
awkrs [OPTIONS] [-f PROGFILE | -e PROGRAM | PROGRAM] [--] [file...]

DESCRIPTION
awkrs runs pattern → action programs over input records like POSIX awk, GNU gawk, and mawk. The CLI accepts the union of POSIX, gawk, and mawk (-W) options.

Execution flows through a Cranelift-JIT bytecode VM with a parallel record processor that activates automatically when the program is parallel-safe and -j selects more than one thread; otherwise execution is sequential.

Source survey claim: among the major public awk implementations (BWK awk, gawk, mawk, goawk, frawk, zawk), awkrs is the first to combine a bytecode VM, a JIT compiler, and a persistent on-disk bytecode cache. frawk has VM + Cranelift/LLVM JIT but recompiles every invocation; gawk's pm-gawk persists script-defined variables, not compiled bytecode.

1. INVOCATION
The program text is taken from the first non-option argument unless -f or -e is given. Remaining non-option arguments are input files; if none are given, standard input is read.

  POSIX

-f PROGFILE, --file=PROGFILE
    Read program source from PROGFILE. Repeatable; sources concatenated.

-F FS, --field-separator=FS
    Set input field separator (FS).

-v var=val, --assign=var=val
    Assign val to var before BEGIN runs. Repeatable.

  GNU program sources

-e PROGRAM, --source=PROGRAM
    Inline program text. Repeatable; sources concatenated.

-i FILE, --include=FILE
    Include awk source file (gawk @include behavior). Repeatable.

  gawk extensions

-b, --characters-as-bytes
    Use byte length for length, substr, index.

-c, --traditional
    Traditional POSIX awk mode.

-C, --copyright
    Print copyright information.

-d[FILE], --dump-variables[=FILE]
    Dump globals after the program runs.

-D[FILE], --debug[=FILE]
    Static rule/function listing. Not gawk's interactive debugger.

-E FILE, --exec=FILE
    Execute program from FILE then exit.

-g, --gen-pot
    Generate gettext .pot output and exit.

-I, --trace
    Trace execution.

-k, --csv
    CSV mode. Sets FS / FPAT and uses a parser aligned with gawk --csv.

-l LIB, --load=LIB
    Load extension library (gawk @load behavior). Repeatable.

-L[VAL], --lint[=VAL]
    Lint checks.

-M, --bignum
    MPFR arbitrary-precision arithmetic.

-n, --non-decimal-data
    Accept hex (0x) and octal (leading 0) input numbers.

-N, --use-lc-numeric
    Respect LC_NUMERIC for sprintf/printf.

-o[FILE], --pretty-print[=FILE]
    Pretty-print parsed program text.

-O, --optimize
    Bytecode optimizations on (default).

-p[FILE], --profile[=FILE]
    Profile execution.

-P, --posix
    Strict POSIX mode.

-r, --re-interval
    Enable interval expressions in regex.

-s, --no-optimize
    Disable bytecode optimizations.

-S, --sandbox
    Disable system, getline | cmd, and other shell-out features.

-t, --lint-old
    Lint against historical awk dialects.

  mawk

-W spec
    Mawk-style options forwarded to the runtime (compat, interactive, sprintf=N, exec=FILE, posix_space, dump, etc.).

  awkrs-specific

-j N, --threads=N
    Worker thread count. Default 1. Activates parallel record processing when the program is parallel-safe (see 9. PARALLEL RECORDS ).

--read-ahead LINES
    Stdin parallel batch size. Default 1024.

--no-jit
    Force the bytecode interpreter even when JIT is available.

--no-cache
    Disable bytecode cache for this invocation.

2. PROGRAM TEXT

  Rules
BEGIN, END, BEGINFILE, ENDFILE, empty pattern, /regex/, expression patterns, range patterns (/a/,/b/ or NR==1,NR==5). The four special patterns must use { ... }; record rules may omit braces for the default { print $0 }.

  Statements
if, while, do-while , for (C-style and for (i in arr)), switch/case/default (gawk-style: no fall-through, regex case /re/), print/printf with >, >>, |, |& redirection, break, continue, next, nextfile, exit, delete, return, getline.

  getline as expression
Returns 1 (read), 0 (EOF), -1 (error), -2 (gawk retryable I/O when PROCINFO[input,"RETRY"] is set). JIT getline failures abort the JIT chunk instead of returning -1/-2.

  Operators
arithmetic, comparison, string concat, ternary, in, ~/!~, ++/-- (prefix/postfix on vars, $n, a[k]), ^/** (right-associative; unary +/-/! bind looser, so -2^2 = -(2^2)). Division by zero (/, compound /=) is a fatal error (division by zero attempted), not infinity.

  Primary /
The lexer may emit / as division when regex_mode is false (e.g. after =). At a primary position / cannot start division, so the parser re-reads as /regex/. In expression context a bare /re/ means $0 ~ /re/ (POSIX), so !/foo/, if (/foo/), x = /foo/ use a match against the current record.

  gawk regexp constants
@/pattern/ yields a regexp value (typeof reports regexp); ~ uses the pattern as a regex.

3. DATA MODEL
Fields, scalars, associative arrays (a[k], a[i,j] with SUBSEP), ARGC/ARGV (set before BEGIN — ARGV[0] is the executable, ARGV[1..] are file paths).

FS is a regex when multi-char. FPAT (gawk-style): non-empty splits by regex match. split/patsplit accept regex as 3rd arg; patsplit 4-arg form populates seps.

POSIX record model: NF = n truncates or extends fields and rebuilds $0 with OFS; $0 = ... re-splits and updates NF.

FS/FPAT from literals: bytecode may store source ... as an internal literal string, but cached_fs sync on read still tracks those assignments.

Whole array in a scalar context (print a, string concat, printf args, ~ operands, etc.) is a fatal runtime error (gawk-style), not a silent empty string.

Scalars: uninitialized variables compare like numeric 0 where POSIX expects dual 0 /

String constants vs input: program string literals are not numeric strings for </<=/>/>= the way $n can be; arithmetic still uses longest-prefix string→number (3.14abc = 3.14).

split("", arr, fs) returns 0 (no empty pseudo-field).

  Records & env
RS is newline by default; one (UTF-8) char is a literal delimiter; RS="" is paragraph mode; multi-char is a gawk regex (RT holds the matched text). FIELDWIDTHS selects fixed-width when non-empty.

ENVIRON, CONVFMT, OFMT, FIELDWIDTHS, IGNORECASE (case-insensitive regex + ==/!= ordering via strcoll), ARGIND, ERRNO, LINT, TEXTDOMAIN, BINMODE. PROCINFO, FUNCTAB, SYMTAB populated in gawk-compatible form.

4. BUILTINS
length , index (empty needle = 1, matching gawk), substr (gawk rule, not POSIX: if start < 1 clamp to 1 and leave length unchanged), intdiv, mkbool, split, sprintf, printf (flags, * and %n$ positional, gawk %', conversions %s %d %i %u %o %x %X %f %e %E %g %G %c %% — %e/%E use signed two-digit exponents; %c uses a string's first character), gsub, sub, match, gensub, isarray, tolower, toupper, int.

Math: sin, cos, atan2, exp, log, sqrt.

rand, srand, systime, strftime (0–3 args), mktime, system, close, fflush, bit ops (and, or, xor, lshift, rshift, compl), strtonum, asort, asorti.

User-defined function with parameter locals. Recursion is capped (256 nested calls in release builds; lower in unit tests) so pathological self-calls fail with an error instead of overflowing the host stack.

5. GAWK EXTENSIONS
@include, @load *.awk , @namespace ... (default identifier prefixing; built-ins exempt), indirect calls (@name(...), @(expr)(...)), /inet/tcp/... and /inet/udp/... client sockets, gettext builtins (bindtextdomain, dcgettext, dcngettext with .mo catalogs via the gettext crate), -M/--bignum MPFR.

6. BYTECODE VM
awkrs compiles AWK programs into a flat bytecode instruction stream and runs them on a stack VM. Short-circuit &&/||, control flow, and range patterns resolve to jump-patched offsets at compile time. The string pool interns variable names and string constants for cheap u32 indexing.

  Peephole fusion
Common sequences combine into single opcodes:

print $N | PrintFieldStdout (zero-alloc field write)
s += $N | AddFieldToSlot (in-place numeric parse)
i = i + 1 / i++ / ++i | IncrSlot (one numeric add, no stack traffic)
s += i between slots | AddSlotToSlot
$1 "," $2 literal concat | ConcatPoolStr
NR++ HashMap-path | IncrVar

  Inline fast paths
Single-rule programs with one fused opcode ({ print $1 }, { s += $1 }) bypass VmCtx entirely.

Memory-mapped files also recognize { gsub( lit with literal pattern: when the needle is absent, the loop writes each line from the mapped buffer with ORS and skips the VM.

7. CRANELIFT JIT
The jit module compiles eligible bytecode chunks to native code using Cranelift with opt_level=speed.

  ABI
(vmctx, slots, field_fn, array_field_add, var_dispatch, field_dispatch, io_dispatch, val_dispatch) -> f64 — opaque vmctx plus seven extern C callbacks (no thread-local lookups).

  Tiered compilation
Chunks count entries and only compile after AWKRS_JIT_MIN_INVOCATIONS (default 3; set 1 for first-entry compile). Set AWKRS_JIT=0 to force the bytecode interpreter.

  Eligible JIT ops
Constants, arithmetic/comparisons, jumps and fused loop tests, slot and HashMap-path scalar ops, field reads (including dynamic $i writes in mixed mode), for-in, asort, asorti, match, sub, gsub, split, patsplit, getline (primary, file, coproc), CallUser, fused print opcodes (including redirects), typeof, and a whitelist of builtins (math, sprintf, printf, strftime, fflush, close, system).

  Value representation
Mixed-mode chunks NaN-box Value in slots; non-mixed chunks keep slots in Cranelift SSA. Unsupported opcodes fall back to the bytecode loop.

8. BYTECODE CACHE
-f script.awk invocations memoize the compiled CompiledProgram to ~/.awkrs/scripts.rkyv — an rkyv-archived shard with mmap + zero-copy ArchivedHashMap lookup on the read path (check_archived_root validation) and flock-serialized atomic-rename writes.

Each entry's inner CompiledProgram blob is bincode (rkyv outer, bincode inner — same architecture as zshrs and strykelang).Repeatrunsskiplex/parse/compileentirely— only the matched entry's blob is decoded.

Invalidation: entries are invalidated on source-file mtime change or when the running awkrs binary is newer than the cached entry. Any rebuild silently rebuilds the cache.

Disable with AWKRS_CACHE=0. The cache only engages for the simple -f script.awk form — inline -e/--source, -E, -i/--include, -l/--load, --debug, --lint, --pretty-print, --gen-pot skip the cache because they need the AST.

9. PARALLEL RECORDS

     ┌─────────────────────────────────────────────┐
     │  WORKER 0  ░░  CHUNK 0   ██ REORDER QUEUE  │
     │  WORKER 1  ░░  CHUNK 1   ██ ──────────────>│
     │  WORKER 2  ░░  CHUNK 2   ██  DETERMINISTIC │
     │  WORKER N  ░░  CHUNK N   ██  OUTPUT STREAM  │
     └─────────────────────────────────────────────┘

Default -j / --threads is 1. Pass a higher value when the program is parallel-safe — static check:

    no range patterns
    no exit, nextfile, delete
    no primary getline
    no pipe/coproc getline
    no asort/asorti
    no indirect calls
    no print/printf redirection
    no cross-record assignments

Records are processed in parallel via rayon and output is reordered to input order within each batch so pipelines stay deterministic.

  File input
Regular files are memory-mapped (memmap2) and scanned with the same RS rules as the sequential path — no read() copy of the whole file.

  Stdin
Stdin parallel chunks up to --read-ahead lines (default 1024) per batch, dispatches to workers, emits in order, then refills.

  Worker model
Workers run the same bytecode VM as the sequential path. The compiled program is shared via Arc<CompiledProgram> (one compile, cheap refcount per worker) with per-worker runtime state.

  Fallback
Non-parallel-safe programs run sequentially with a warning when -j > 1. Programs that use primary getline (including in BEGIN) also run sequentially for file input. END only sees post- BEGIN global state — record-rule mutations from parallel workers are not merged.

10. BENCHMARKS
Combat metrics on representative workloads (1 million lines unless noted). awkrs ties or beats every other awk implementation surveyed on most workloads. See README [0x06] BENCHMARKS for the full table with timings. Highlights:

{ print $1 } throughput | awkrs (fused PrintFieldStdout)
CPU-bound BEGIN (no input) | awkrs / frawk near-tie
Sum first column | awkrs (AddFieldToSlot)
Regex filter (no matches) | awkrs / mawk
Associative array build | awkrs / gawk
gsub no-match (mmap fast path) | awkrs (mapped-buffer write, VM skipped)

11. ENVIRONMENT

AWKPATH
    Search path for -l and @include. Default ..

AWKLIBPATH
    Search path for @load.

AWKRS_CACHE
    Set to 0 to disable the bytecode cache.

AWKRS_JIT
    Set to 0 to force the bytecode interpreter.

AWKRS_JIT_MIN_INVOCATIONS
    JIT tier-up threshold (default 3).

GAWK_READ_TIMEOUT
    Default read timeout (ms) used as the fallback for PROCINFO[input,"READ_TIMEOUT"].

LC_NUMERIC
    Locale decimal radix and thousands grouping when -N is set; affects sprintf/printf/print output only, not $n/$0 parsing.

12. EXIT STATUS

0   Successful completion.

1   Error during program execution or invalid usage. Errors are written to stderr as awkrs: <command>: <reason>.

n   The program may exit with any status by calling exit n.

13. FILES

~/.awkrs/scripts.rkyv
    Persistent bytecode cache shard. rkyv outer + bincode inner; mmap+validate on read, flock-serialized atomic rename on write.

/usr/local/share/man/man1/awkrs.1
    Short reference man page.

/usr/local/share/man/man1/awkrsall.1
    This man page.

14. EXAMPLES

Print the second whitespace-separated field:
    awkrs '{ print $2 }' input.txt

Sum a CSV column with quoted-field handling and four worker threads:
    awkrs -k -j4 '{ s += $3 } END { print s }' data.csv

Memoize compiled bytecode on disk:
    awkrs -f myscript.awk input.txt
    # second invocation skips lex / parse / compile entirely

Force the bytecode interpreter (no JIT):
    AWKRS_JIT=0 awkrs '{ print $2 }' input.txt

First-entry JIT (tier-up threshold = 1):
    AWKRS_JIT_MIN_INVOCATIONS=1 awkrs -f hot.awk huge.log

CSV mode with named fields:
    awkrs -k '{ print $1, $5 }' data.csv

TCP client socket:
    awkrs 'BEGIN { "/inet/tcp/0/example.com/80" |& getline line; print line }'

Indirect function call:
    awkrs 'function f(x) { return x*2 } BEGIN { name = "f"; print @name(21) }'

gawk regexp constant:
    awkrs 'BEGIN { r = @/^[0-9]+$/; print typeof(r) }'

Parallel-safe sum across 8 threads:
    awkrs -j8 '{ s += $1 } END { print s }' big.log

BEGINFILE hook:
    awkrs 'BEGINFILE { print "opening", FILENAME } /pattern/ { print }' *.log

--debug static listing:
    awkrs --debug=program.lst -f myscript.awk

MPFR arbitrary precision:
    awkrs -M 'BEGIN { PREC = 200; print 1 / 7 }'

Locale-aware printf:
    LC_NUMERIC=fr_FR.UTF-8 awkrs -N 'BEGIN { printf("%.2f\n", 1234.5) }'

15. PARITY GAPS
awkrs implements the union of POSIX, gawk, and mawk. Remaining gaps are documented in the project compatibility matrix (docs/COMPATIBILITY.md). Notable items:

•   -D --debug emits a static rule/function listing; it is not gawk's interactive debugger (no breakpoints, no step / next).

•   -c --traditional is stored on the runtime; effect today is minimal.

•   Some PROCINFO[input,...] keys (e.g. RETRY, READ_TIMEOUT) are honored only on retryable I/O paths.

•   Cross-record state mutations in parallel mode are not merged into END; the static safety check refuses such programs unless -j 1 is used.

SEE ALSO
awkrs(1), awk(1), gawk(1), mawk(1), strykeall(1), zshrsall(1), lsofrsall(1), temprsall(1)

AUTHOR
Jacob Menke <https://github.com/MenkeTechnologies>

BUGS
Report bugs at https://github.com/MenkeTechnologies/awkrs/issues.