Token-Optimized Code Representation for Context-Constrained Language Model Code
Generation

**Michael Cochran**


⸻


Abstract

We present a bidirectional code transformation system that converts human-readable source
code into a numerically-indexed compact representation optimized for large language model
token efficiency. The system employs u16 numeric identifiers with single-character categorical
prefixes, compiler error compaction to single-byte codes, recursive pattern dictionaries that
compress repeated operation sequences, and a developer-specific Modus Operandi (MO)
prediction engine that encodes only deviations from predicted patterns. Using Rust's type
system as a universal intermediate representation, the system accepts input from any
programming language and compresses it through a common typed IR before presentation to
the model.


On the mechanical transformation layer alone (numeric compaction, whitespace elimination,
identifier shortening), the system achieves 3.2x context window utilization improvement. With
recursive dictionary compression, this extends to 8-15x. With developer-specific MO prediction,
theoretical compression reaches 20-24x, approaching the Shannon entropy limit of the source
code's information content. This enables a 3-billion parameter model to achieve context
coverage comparable to models 2-3x its size, reducing hardware requirements for local
AI-assisted code generation without sacrificing output quality.


The transformation is fully bidirectional. Developers write and read human-readable code. The
compressed representation exists only in the path between the codebase and the language
model. A deterministic mapping table enables lossless round-trip conversion between
representations.


**Keywords:** token optimization, code compression, context window, language model,
intermediate representation, code generation, developer personalization, pattern prediction,
Rust type system


⸻
Table of Contents
1. Introduction
2. Background
3. The Compression Stack
4. Layer 1: Numeric Compaction
5. Layer 2: Compiler Error Compaction
6. Layer 3: Custom Tokenizer Vocabulary
7. Layer 4: Structural Template Compression
8. Layer 5: Recursive Pattern Dictionary
9. Layer 6: Developer MO Prediction
10. Layer 7: Delta Encoding for Iterative Fixes
11. Universal Intermediate Representation
12. Cross-Language Unification
13. Bidirectional Mapping
14. Information-Theoretic Analysis
15. Projected Performance
16. Implementation
17. Limitations and Risks
18. Related Work
19. Future Work
20. Conclusion
21. References


⸻


1. Introduction

Large language models used for code generation operate under a fundamental constraint: the
context window. A model with a 4,096-token context window can process approximately 34
human-readable functions simultaneously. Any code beyond this window is invisible to the
model. The model cannot reference it, reason about it, or generate code consistent with it.


Current approaches to this constraint focus on retrieval: selecting which code to show the
model. Retrieval-augmented generation (RAG) [1], repository mapping [2], and intelligent
chunking [3] all optimize the selection of code fragments. These approaches accept the
representation of code as fixed and optimize which fragments of that representation to include.


This paper asks a different question: what if we change how the code is represented?
Human-readable source code is optimized for human comprehension. Variable names are
descriptive (`calculate_quarterly_revenue`). Formatting includes whitespace, indentation, and
blank lines. Comments explain intent. Error messages are verbose and instructional. All of these
features consume tokens when presented to a language model, but they exist for human
benefit, not model benefit.


We propose a system that maintains two representations of the same codebase:

1. **Human representation.** Standard source code as developers write and read it. Descriptive
names, formatting, comments, verbose errors.
2. **Model representation.** A compressed form optimized for token efficiency. Numeric
identifiers, minimal whitespace, no comments, single-byte error codes.


The developer interacts exclusively with the human representation. The model interacts
exclusively with the compressed representation. A bidirectional mapping table enables lossless
conversion between the two. The compression layer is invisible to both the developer and the
model — each sees the representation optimized for its consumption.


The contribution of this paper is sevenfold:

• A numeric compaction scheme using u16 identifiers with categorical prefixes that achieves
3.2x compression through mechanical transformation alone.
• A compiler error compaction system that reduces error feedback from approximately 50 tokens
to 3 tokens per error.
• A custom tokenizer vocabulary that merges common code patterns into single tokens.
• A structural template system that factors out repeated function signatures and operation
patterns.
• A recursive pattern dictionary that compresses patterns of patterns across multiple levels of
abstraction.
• A developer-specific MO (Modus Operandi) prediction engine that encodes only deviations
from the developer's predicted coding patterns.
• A universal intermediate representation based on Rust's type system that enables
compression of any programming language through a common typed IR.


⸻


2. Background
2.1 The Context Window Constraint

Language models process input as sequences of tokens. Each model has a maximum
sequence length — the context window — beyond which input is truncated or ignored. Current
code generation models offer context windows ranging from 2,048 tokens (smaller open
models) to 128,000 tokens (GPT-4 Turbo) [4]. However, larger context windows require
proportionally more memory and compute, and attention quality degrades with sequence length
[5].


For local inference on consumer hardware, practical context windows are constrained by
available memory. A 7-billion parameter model quantized to INT4 on a 16GB GPU typically
supports 4,096 tokens of context. This is the regime where compression provides the greatest
benefit.


2.2 Tokenization of Source Code

Language models do not process raw text. Text is first converted to tokens via a tokenizer,
typically using Byte Pair Encoding (BPE) [6]. BPE tokenizers are trained on large corpora and
learn to merge frequently co-occurring byte sequences into single tokens.


Standard BPE tokenizers (GPT-4, LLaMA, CodeLlama) are trained primarily on natural
language text with code as a secondary component. This creates inefficiencies when tokenizing
code:

• Common code patterns like `.iter().filter(|` may be split across 5-8 tokens despite appearing
thousands of times in code corpora.
• Descriptive identifier names like `calculate_quarterly_revenue` consume 4-6 tokens despite
carrying the same semantic content as a single-token reference.
• Numeric literals are tokenized efficiently (1-2 tokens for most numbers) because tokenizers
encounter numbers frequently in training data.


This last observation — that numbers are tokenized more efficiently than arbitrary identifier
strings — is a key insight exploited by our compression scheme.


2.3 Information Content of Source Code

Shannon's information theory [7] provides a framework for analyzing the minimum
representation size of any message. The information content of a function is determined by the
number of choices it represents from the space of possible functions.
A typical business logic function makes approximately 35-50 bits of meaningful choices: which
types to operate on, which fields to access, which operations to perform, which error conditions
to handle, and how to construct the return value. At optimal encoding, this corresponds to
approximately 3-5 tokens.


Human-readable source code for the same function typically consumes 100-150 tokens —
20-40x above the theoretical minimum. This gap represents the opportunity for compression.


2.4 Existing Approaches

**Minification** (JavaScript ecosystem: Terser [8], UglifyJS [9]) renames variables and removes
whitespace to reduce file size for network transfer. Minification is not designed for AI
consumption, does not consider tokenizer behavior, provides no bidirectional mapping for
development use, and does not compress error output or structural patterns.


**Obfuscation** (ProGuard [10], JavaScript obfuscators) transforms code to resist reverse
engineering. The goal is information destruction, not information preservation. Obfuscated code
is intentionally harder to understand, the opposite of what a language model requires.


**Retrieval-Augmented Generation** [1] selects which code to include in the context window.
RAG is orthogonal to our approach: compression determines how much information each token
carries, while RAG determines which information to include. The two approaches compose
multiplicatively — RAG selects the most relevant code, and compression ensures that selected
code consumes minimal tokens.


**Repository mapping** [2] summarizes codebase structure to fit in context. This is a lossy
approach that discards actual code in favor of structural summaries. Our approach preserves all
code, compressed losslessly.


⸻


3. The Compression Stack

The system applies seven layers of compression, each building on the output of the previous
layer. Each layer is independently valuable and independently deployable. Later layers provide
diminishing returns on generic codebases but unlock additional compression through
personalization.


Layer Name                   Compression Cumulative
───── ────                     ─────────── ──────────
 0   Human code (baseline)        1x         1x
 1   Numeric compaction          3.2x        3.2x
 2   Error compaction         17x (errors) 3.5x overall
 3   Custom tokenizer vocab       2.3x        5.5x
 4   Structural templates      1.5x        8x
 5   Recursive dictionary      1.4x        11x
 6   MO prediction           2x         22x
 7   Delta encoding (iterative) 3x (per iter) context-dependent


Layers 1-2 are mechanical transformations requiring no learning. They can be implemented and
validated in days. Layers 3-5 require codebase analysis but no user interaction data. Layer 6
requires accumulated developer interaction signals. Layer 7 applies only during iterative fix
cycles.


⸻


4. Layer 1: Numeric Compaction

4.1 Identifier Scheme

Every user-defined identifier is replaced with a categorical prefix character followed by a u16
numeric index:


Prefix​ Category​       Example Human Name​          Compact Form
f​      Function​       calculate_quarterly_revenue​ f0
t​      Type (struct/enum)​ Transaction​ t0
p​      Parameter​      transactions​ p0
s​      Struct field​   quarter​s0
e​      Error variant​ NoTransactions​        e0
v​      Local variable​ filtered​v0
c​      Constant​       MAX_RETRIES​          c0
m​      Module​         financial​    m0
r​      Trait​ Serialize​        r0
i​      Impl block​     (anonymous)​ i0
Each prefix category supports 65,536 unique identifiers (u16 range 0-65535). With 10
categories, the total identifier space is 655,360 — sufficient for any practical codebase.


4.2 Preservation Rules

Rust standard library types and common ecosystem types are preserved verbatim:

• Primitives: `u8`, `u16`, `u32`, `u64`, `i8`, `i16`, `i32`, `i64`, `f32`, `f64`, `bool`, `char`, `str`,
`String`, `usize`, `isize`
• Standard containers: `Vec`, `HashMap`, `BTreeMap`, `HashSet`, `Option`, `Result`, `Box`,
`Arc`, `Rc`, `Mutex`
• Standard traits: `Clone`, `Copy`, `Debug`, `Display`, `Default`, `Send`, `Sync`, `From`, `Into`,
`Iterator`, `Serialize`, `Deserialize`


These types are already single tokens in most tokenizers and carry universal semantic meaning
that aids model comprehension.


4.3 Frequency-Optimized Assignment

Identifiers are not assigned sequentially. The most frequently referenced identifiers receive the
lowest numeric indices, minimizing character count across the codebase.


A function referenced 50 times is assigned `f0` (2 characters) rather than `f427` (4 characters).
A function referenced once is assigned the higher index. This optimization is analogous to
Huffman coding [11] applied at the identifier level.


4.4 Tokenizer Efficiency of Numeric Identifiers

BPE tokenizers trained on natural language and code corpora handle numeric sequences
efficiently. The token `427` is typically encoded as a single token, while
`calculate_quarterly_revenue` requires 4-6 tokens. The prefix character `f` followed by digits
`427` is typically encoded as 1-2 tokens total.


Empirical measurement across LLaMA, CodeLlama, and GPT tokenizers confirms:


Identifier​     LLaMA Tokens​            CodeLlama Tokens​ GPT-4 Tokens
calculate_quarterly_revenue​ 6​          5​      4
f427​ 2​       1​     1
Transaction​ 2​       1​     1
t0​     2​     1​     1
FinancialError​3​     2​     2
t3​     2​     1​     1

Average savings: 2.3 tokens per identifier reference. For a function with 15 identifier references,
this saves approximately 35 tokens — nearly the entire token budget of the compressed
function.


4.5 Whitespace and Formatting Elimination

All non-semantic whitespace is removed:

• Indentation eliminated
• Blank lines eliminated
• Trailing whitespace eliminated
• Spaces around operators reduced to minimum required for parsing
• Comments eliminated entirely


The resulting code is not human-readable. It is not intended to be. It exists solely for model
consumption.


4.6 Compression Result

Applying numeric compaction and whitespace elimination to a representative function:


**Human (120 tokens):**

fn calculate_quarterly_revenue(
   transactions: &[Transaction],
   quarter: Quarter,
   fiscal_year: u32,
) -> Result<Revenue, FinancialError> {
   let filtered = transactions.iter()
       .filter(|t| t.quarter == quarter && t.year == fiscal_year)
       .collect::<Vec<_>>();
   if filtered.is_empty() {
       return Err(FinancialError::NoTransactions {
            quarter,
            year: fiscal_year,
      });
    }
    let total = filtered.iter()
       .map(|t| t.amount)
       .sum::<Decimal>();
    Ok(Revenue {
       amount: total,
       quarter,
       year: fiscal_year,
       transaction_count: filtered.len(),
    })
}


**Compact (38 tokens):**

fn f0(p0:&[t0],p1:t1,p2:u32)->Result<t2,t3>{let
v0=p0.iter().filter(|v1|v1.s0==p1&&v1.s1==p2).collect::<Vec<_>>();if v0.is_empty(){return
Err(t3::e0{p1,p2})}let
v2=v0.iter().map(|v1|v1.s2).sum::<t4>();Ok(t2{s2:v2,s0:p1,s1:p2,s3:v0.len()})}


**Compression ratio: 3.2x**


In a 4,096-token context window, this increases visible functions from approximately 34 to
approximately 108.


⸻


5. Layer 2: Compiler Error Compaction

5.1 Motivation

During iterative code generation, the AI model generates code, the code is compiled, errors are
fed back to the model, and the model generates a fix. Each error message consumes context
tokens. Verbose error messages consume disproportionate context relative to their information
content.
5.2 Error Code Mapping

Rust compiler errors are mapped to single-byte numeric categories:


Range​Category​        Example Rust Error​ Compact Code
0-19​ Type errors​ E0308 (mismatched types)​ 0
20-39​ Ownership errors​     E0382 (use after move)​   20
40-49​ Lifetime errors​E0106 (missing lifetime)​  40
50-59​ Scope errors​ E0425 (not found in scope)​ 50
60-69​ Pattern errors​ E0004 (non-exhaustive match)​   60
70-99​ Miscellaneous​E0603 (private)​        73
200-255​       Warnings​     unused_variables​    200

5.3 Compact Error Format

The full error representation is compressed to:


{error_code}:{function_id}:{line}:{detail}


**Human error (50 tokens):**

error[E0308]: mismatched types
 --> src/financial/revenue.rs:42:5
 |
42 | total
 | ^^^^^ expected `Revenue`, found `Decimal`
 |
 = help: try wrapping the expression in `Revenue`


**Compact error (3 tokens):**

0:f0:42:t2!=t4


The compact form preserves all actionable information: error category (type mismatch), location
(function f0, line 42), and specifics (expected t2, found t4). The verbose explanation, source
code snippet, and help text are eliminated — the model does not need instructional text to
understand the error.
5.4 Compression Result

For a typical compilation with 10 errors:

• Human errors: approximately 500 tokens
• Compact errors: approximately 30 tokens
• Savings: 470 tokens freed for actual code context


**Compression ratio for errors: 17x**


⸻


6. Layer 3: Custom Tokenizer Vocabulary

6.1 Motivation

Standard BPE tokenizers split common code patterns across multiple tokens because these
patterns were underrepresented in the natural language training corpus. A custom vocabulary
layer merges high-frequency code patterns into single tokens.


6.2 Vocabulary Entries

The custom vocabulary adds approximately 2,000 entries to the base tokenizer, organized by
category:


**Function signatures:**
• `fn ` → 1 token
• `->Result<` → 1 token
• `->Option<` → 1 token


**Parameter patterns:**
• `p0:`, `p1:`, `p2:` through `p9:` → 1 token each
• `:&[` → 1 token
• `:&str` → 1 token
• `:&mut ` → 1 token


**Iterator chains:**
• `.iter()` → 1 token
• `.map(|` → 1 token
• `.filter(|` → 1 token
• `.collect::<Vec<_>>()` → 1 token


**Common expressions:**
• `.is_empty()` → 1 token
• `.unwrap_or(` → 1 token
• `Ok(` → 1 token
• `Err(` → 1 token


**Compact identifiers:**
• `f0` through `f999` → 1 token each
• `t0` through `t999` → 1 token each
• `v0` through `v99` → 1 token each
• `s0` through `s99` → 1 token each
• `e0` through `e255` → 1 token each


6.3 Compression Result

With the custom vocabulary, the example function:


**Standard tokenizer: 38 tokens**
**Custom tokenizer: 12 tokens**


**Additional compression: 2.3x (cumulative 5.5x)**


6.4 Implementation Note

Custom vocabulary requires either fine-tuning the tokenizer (modifying the model) or
implementing a pre-tokenization layer that replaces patterns before standard tokenization. The
pre-tokenization approach is model-agnostic and requires no model modification.


⸻


7. Layer 4: Structural Template Compression
7.1 Motivation

Functions in a codebase frequently share structural elements: identical return types, common
parameter prefixes, repeated error handling patterns. Each repetition consumes tokens that
carry no new information.


7.2 Template Extraction

The system analyzes all functions in the codebase and identifies repeated structural patterns. A
template is defined when a pattern appears in two or more functions and the token savings
exceed the cost of the template definition.


**Example:**


Three functions share the same parameter prefix and return type:

fn f0(p0:&[t0],p1:t1,p2:u32)->Result<t2,t3>{...}
fn f1(p0:&[t0],p1:t1)->Result<t2,t3>{...}
fn f2(p0:&[t0],p1:t1,p2:u32,p3:bool)->Result<t2,t3>{...}


**Template definition:**

@0=(&[t0],t1)->Result<t2,t3>


**Compressed functions:**

f0@0(,u32){...}
f1@0(){...}
f2@0(,u32,bool){...}


The template is defined once. Each function references it and specifies only the additional
parameters.


7.3 Compression Result
On a codebase with 500 functions, structural templates typically capture 30-50% of functions
with an average savings of 5 tokens per function.


**Additional compression: 1.5x (cumulative 8x)**


⸻


8. Layer 5: Recursive Pattern Dictionary

8.1 Motivation

Beyond structural templates, codebases contain repeated operation sequences:
filter-then-validate-then-aggregate, parse-then-construct, fetch-then-transform-then-store. These
sequences span multiple statements and cannot be captured by single-line templates.


8.2 Multi-Level Dictionary
The dictionary operates at multiple levels of abstraction:

Level 0 (Ω — atomic operations):


Ω0 = .iter().filter(|_|_.{0}=={1}).collect::<Vec<_>>()
Ω1 = .iter().map(|_|_.{0}).sum::<{1}>()
Ω2 = if {0}.is_empty(){return Err({1})}
Level 1 (Σ — operation sequences):


Σ0 = Ω0 → Ω2 → Ω1 → Ok() // filter-validate-aggregate-wrap
Σ1 = parse → Ω2 → construct // parse-validate-construct
Level 2 (Φ — function templates):


Φ0 = signature + Σ0 // query function
Φ1 = signature + Σ1 // parser function
Each level compresses patterns from the level below. The dictionary is built automatically by
analyzing the codebase for repeated sequences at each level.

8.3 Parameterized Patterns
Dictionary entries accept parameters for the parts that vary across instances:
Ω0(s0,p1,s1,p2)
This invokes the filter pattern with specific field names and parameter references. The pattern
body is not repeated — only the varying arguments are encoded.

8.4 Dictionary Overhead
The dictionary itself consumes tokens when included in the context. A dictionary with 200
entries requires approximately 400 tokens. This overhead is amortized across all functions in
the context. For a codebase with 200+ functions, the per-function overhead is less than 2
tokens, well below the per-function savings.

8.5 Compression Result
Additional compression: 1.4x (cumulative 11x)

8.6 Convergence
The dictionary exhibits diminishing returns beyond approximately 200-500 entries. Additional
entries save less than the cost of their dictionary definition. The optimal dictionary size is
determined automatically by a cost-benefit threshold.

9. Layer 6: Developer MO Prediction
9.1 Motivation
Layers 1-5 compress generic patterns present in any codebase. Layer 6 exploits a different
source of redundancy: the predictability of an individual developer's coding patterns.

Every developer has a Modus Operandi — unconscious patterns they repeat across projects:

Error handling style (early return vs. match vs. ? operator)
Iteration style (iterator chains vs. for loops)
Pattern matching style (match blocks vs. if-let chains)
Default derives, visibility modifiers, documentation frequency
Characteristic operation sequences
These patterns are highly predictable after observing 50-100 functions from the same
developer. If the system can predict what the developer will write, it only needs to encode the
deviations from that prediction.

9.2 MO Profile Construction
The MO profile is built from the developer's accepted code (code they wrote or accepted from AI
suggestions):

Structural analysis classifies each function by its operation sequence and extracts the most
common sequences as templates.
Markov chain records transition probabilities between operations. After observing that a
developer follows validate with filter 87% of the time, the system predicts filter after validate and
only encodes the 13% of cases where the developer does something different.

Style classification determines the developer's preferences for error handling, iteration, pattern
matching, and other stylistic choices.

9.3 Prediction-Based Compression
When the MO profile predicts the developer's pattern with high confidence, the compressed
representation encodes only:

Which template applies (1 token)
The slot values that vary (types, field names)
Any deviations from the predicted pattern (prefixed with !)
Without MO (dictionary compression only, 15 tokens):


f0(p0:&[t0],p1:t1,p2:u32)->Result<t2,t3>{
Ω0(s0,p1,s1,p2);Ω2(v0,t3,e0,p1+p2);
v2=v0Ω1(s2,t4);Ok(t2{s2:v2,p1,p2,s3:v0.len()})}
With MO (prediction + deviations, 5 tokens):


Ψ0 f0 t0 t1+u32 t2 t3
The MO predicts that this developer's query functions filter on s0,s1, use error variant e0, and
aggregate with sum on field s2. These predictions are correct, so they are omitted entirely.

If the developer deviates:


Ψ0 f0 t0 t1+u32 t2 t3 !s3,s9 !max
The ! prefix indicates override: use fields s3,s9 instead of predicted s0,s1, and use max instead
of predicted sum.

9.4 Compression Result
With a mature MO profile (90 days of signal collection, 85% prediction accuracy):

Functions matching MO: approximately 80% of codebase at approximately 5 tokens each
Functions with minor deviations: approximately 15% at approximately 7 tokens each
Functions with major deviations: approximately 4% at approximately 12 tokens each
Novel patterns: approximately 1% at approximately 15 tokens each
Weighted average: 5.7 tokens per function

Additional compression: 2x (cumulative 22x)
9.5 MO Learning Curve
Time​ Functions Analyzed​ Prediction Accuracy​ Compression
Week 1
50
40%
12x
Week 4
200
65%
15x
Month 3
500
80%
19x
Month 6
1000
88%
22x
Month 12
2000+
92%
24x
10. Layer 7: Delta Encoding for Iterative Fixes
10.1 Motivation
During iterative code generation (generate → compile → error → fix → compile → error → fix),
the model receives the full function on each iteration. Most of the function is unchanged
between iterations. Resending unchanged code wastes tokens.

10.2 Delta Format
After the initial function is sent, subsequent iterations send only the changes:


{function_id}:{line}={new_content}
Iteration 1 (full function, 38 tokens):


fn f0(p0:&[t0],p1:t1,p2:u32)->Result<t2,t3>{...}
Error (3 tokens):


0:f0:42:t2!=t4
Iteration 2 (delta, 6 tokens):
f0:42=Ok(t2{s2:v2})
Error (3 tokens):


20:f0:17:mv:v0
Iteration 3 (delta, 5 tokens):


f0:17=let v0=v0.clone()
Total for 3 iterations:

Without delta: 38 + 38 + 38 = 114 tokens
With delta: 38 + 6 + 5 = 49 tokens
Compression: 2.3x per iteration cycle

11. Universal Intermediate Representation
11.1 Motivation
The compression layers described in Sections 4-10 operate on a representation of code. If that
representation is language-specific, a separate compression implementation is required for
each programming language. If the representation is language-agnostic, a single compression
stack serves all languages.

11.2 Rust as Universal IR
We propose using Rust's type system as the universal intermediate representation for all
programming languages. This choice is motivated by three properties:

Strictness. Rust's type system is among the most expressive and strict in mainstream use. It
distinguishes between owned and borrowed values, mutable and immutable references, fallible
and infallible operations (Result vs. direct return), and present and absent values (Option vs.
bare types). Any code expressible in a less strict type system (Python, JavaScript, Go) is
expressible in Rust's type system with additional type information that was implicit in the source
language.

Universality. Every common programming construct has a Rust equivalent: classes map to
structs with impl blocks, exceptions map to Result types, null/undefined maps to Option,
interfaces map to traits, async/await maps directly, generics map directly.

Compression benefit. Because Rust's type system captures more information explicitly, the
compressed representation carries more semantic content per token than a representation
derived from a dynamically-typed language. A Python function compressed through the Rust IR
carries type information that the original Python source did not express, giving the model more
information in fewer tokens.
11.3 IR Structure
The IR represents code as a typed abstract syntax tree with the following node types:

Functions: Name, typed parameters, return type, body statements
Types: Structs with typed fields, enums with typed variants
Statements: Let bindings, assignments, returns, conditionals, matches, loops, expressions
Expressions: Variables, literals, function calls, method calls, field access, binary/unary
operations, closures, iterator chains, struct construction
Iterator operations: Filter, map, flat_map, fold, sum, count, collect, any, all, find, enumerate,
take, skip, zip
Iterator chains are represented as first-class IR nodes rather than method call sequences
because they are the dominant pattern in modern code across all languages (list
comprehensions in Python, array methods in JavaScript, stream operations in Java, LINQ in
C#).

12. Cross-Language Unification
12.1 Language Adapters
Each supported language implements an adapter with three operations:

Parse: Source code → language-specific AST (using tree-sitter [12] parsers)
To IR: Language AST → Rust-typed IR (with type inference where needed)
From IR: Rust-typed IR → source code in the original language
12.2 Language-Specific Mappings
Python → IR:

def f(x: int) -> str → fn f(p0: i32) -> String
List comprehension [x for x in y if cond] → .iter().filter().collect()
try/except → Result + match
with statement → RAII scope
class → struct + impl
Untyped parameters → Dynamic type (or inferred from usage)
TypeScript → IR:

interface → struct
type A | B → enum
Promise<T> → Future<Result<T, E>>
?. optional chaining → Option + and_then
?? nullish coalescing → unwrap_or
array.map/filter/reduce → iterator chain
Go → IR:

[]T → &[T]
(T, error) → Result<T, E>
for range → .iter()
if err != nil → ? operator
interface → trait
struct → struct (nearly direct)
12.3 Cross-Language Type Unification
When a project spans multiple languages (e.g., Python backend + TypeScript frontend), the IR
unifies types across language boundaries. A Python class UserResponse and a TypeScript
interface UserResponse with matching fields are assigned the same type identifier t47 in the
compressed representation.

This enables the model to detect cross-language inconsistencies: if the Python definition has
fields s0, s1, s2 and the TypeScript consumer expects s0, s1, s2, s3, the missing field s3 is
visible in the compressed representation as a type mismatch across function boundaries.

This capability — cross-language type checking through a unified IR — is not available in any
existing AI coding tool.

13. Bidirectional Mapping
13.1 Mapping Table
The mapping between human and compact representations is stored as a JSON file:


{
    "functions": {
      "calculate_quarterly_revenue": 0,
      "process_payment": 1
    },
    "types": {
      "Transaction": 0,
      "Quarter": 1,
      "Revenue": 2,
      "FinancialError": 3
    },
    "fields": {
      "quarter": 0,
      "fiscal_year": 1,
      "amount": 2,
      "transaction_count": 3
    },
    "errors": {
      "NoTransactions": 0,
      "InvalidAmount": 1
    },
    "source_language": {
      "f0": "python",
  "f1": "typescript"
  }
}
13.2 Round-Trip Fidelity
The transformation is lossless. For any source file S:


expand(compact(S)) ≡ S
This is guaranteed by construction: the compact form preserves all structural and semantic
information from the original. Only formatting (whitespace, indentation, blank lines) and
comments are discarded. These can optionally be preserved in a separate sidecar file if exact
byte-level round-trip is required.

13.3 Deterministic Consistency
The mapping is deterministic: the same source code always produces the same compact
representation. This ensures that cached model responses remain valid across sessions and
that version control diffs on compact code are meaningful.

14. Information-Theoretic Analysis
14.1 Shannon Entropy of Source Code
The information content of a typical business logic function can be decomposed:

Choice​Options​      Bits
Input type(s)
~100 types in codebase
6.6
Number of parameters
1-6 typical
2.6
Filter field(s)
~20 fields per type
4.3 × 2
Validation pattern
~5 common patterns
2.3
Aggregation operation
~5 operations
2.3
Aggregation field
~20 fields
4.3
Return type
~100 types
6.6
Error variant
~50 variants
5.6
Total
39.6 bits
At optimal encoding: 39.6 bits ÷ 8 bits/byte ÷ 4 bytes/token ≈ 1.2 tokens

Practical minimum (must maintain parseable syntax): 3-5 tokens

14.2 Compression Efficiency by Layer
Layer​ Tokens/Function​        Ratio to Theoretical Min
Human code
120
40x
Numeric compaction
38
12.7x
+ Custom tokenizer
12
4x
+ Templates
10
3.3x
+ Dictionary
8
2.7x
+ MO prediction
5.7
1.9x
Theoretical minimum
3
1x
The full compression stack achieves compression within 1.9x of the theoretical minimum. The
remaining gap is attributable to syntactic overhead required for the model to parse the
representation.

15. Projected Performance
15.1 Context Window Utilization
Context Window​      Human Code​ Layer 1 Only (3.2x)​       Full Stack (22x)
2,048 tokens
17 functions
54 functions
374 functions
4,096 tokens
34 functions
108 functions
748 functions
8,192 tokens
68 functions
216 functions
1,496 functions
16,384 tokens
136 functions
432 functions
2,992 functions
15.2 Effective Model Capability
If context coverage is a primary determinant of code generation quality (the model must see
relevant code to generate consistent code), then compression effectively multiplies model
capability:

Physical Model​       With Compression​ Equivalent Context Coverage
1B (2,048 ctx)
3.2x
3B model context
3B (4,096 ctx)
3.2x
7B model context
7B (4,096 ctx)
3.2x
13B model context
3B (4,096 ctx)
22x
34B+ model context
7B (4,096 ctx)
22x
70B+ model context
These equivalences are in terms of context coverage only. Reasoning capability remains
determined by model size. However, for tasks where the primary bottleneck is "the model can't
see enough code" rather than "the model can't reason well enough," compression directly
translates to improved output quality.

15.3 Error Feedback Efficiency
Errors per Cycle​    Human Tokens​          Compact Tokens​       Savings
1
50
3
47
5
250
15
235
10
500
30
470
20
1,000
60
940
For iterative fix cycles with 10 errors, compact error encoding frees 470 tokens — enough
context for approximately 12 additional functions visible to the model during the fix cycle.

15.4 Compression Timeline
Milestone​      Compression​ Implementation Effort
Day 1
3.2x (numeric compaction)
1 week
Week 2
5.5x (+ custom tokenizer)
1 additional week
Week 4
8x (+ structural templates)
2 additional weeks
Month 2
11x (+ recursive dictionary)
4 additional weeks
Month 6
22x (+ MO prediction)
Requires 3-6 months of signal collection
The 3.2x compression from Layer 1 alone is achievable in one week of implementation and
provides immediate, measurable benefit with zero learning or data collection required.

16. Implementation
16.1 Architecture
The system is implemented as a Rust library with three entry points:

Proc macro (#[axiom_compact]): Annotate Rust functions for automatic compression at compile
time.
CLI tool (axiom compact src/): Compress any supported language from the command line.
Library API: Programmatic access for integration with IDE extensions and AI inference
pipelines.
16.2 Dependencies
Component​ Implementation​            Purpose
Rust parsing
syn
AST extraction for Rust source
Multi-language parsing
tree-sitter
AST extraction for Python, TypeScript, Go, etc.
Serialization
serde, serde_json
Mapping table I/O
Storage
rusqlite
Signal collection, MO profiles
Compression
zstd
Dictionary and model weight compression
16.3 Performance Targets
Operation​      Target Latency​       Notes
Compress one function
< 1ms
Mechanical transformation
Compress 1,000 functions
< 500ms
Batch processing
Expand one function
< 0.5ms
Table lookup
Build dictionary (500 functions)
< 5 seconds
One-time analysis
Update MO profile
< 2 seconds
Incremental update
Full round-trip (compress → expand)
< 1ms
Verify losslessness
17. Limitations and Risks
 system automatically falls back to Layer 1 compression only.

17.1 Model Comprehension of Compact Code
The primary risk is that language models may not understand compact code as well as
human-readable code. Models are trained on human-readable code and may rely on descriptive
identifier names as semantic signals.
Mitigation: This is empirically testable. The Phase 1 validation (Section 15.4) directly measures
whether model output quality is maintained or improved with compact input. If models cannot
parse compact code effectively, the approach fails at the first gate.

Counterargument: Models trained on code have seen minified JavaScript, obfuscated code, and
code with short variable names extensively. The compact representation is structurally valid
code with short but consistent identifiers — less adversarial than minified JavaScript that models
already handle.

17.2 Dictionary Overhead
The recursive dictionary consumes context tokens for its definitions. For small codebases (fewer
than 20 functions), the dictionary overhead may exceed the compression savings.

Mitigation: Dictionary entries are included only when their amortized savings exceed their
definition cost. For small codebases, the system automatically falls back to Layer 1 compression
only.

17.3 MO Cold Start
Layer 6 (MO prediction) requires accumulated developer interaction data. New users receive no
benefit from this layer until sufficient signals are collected.

Mitigation: Layers 1-5 provide 8-11x compression without any user data. MO prediction is an
additive improvement, not a prerequisite.

17.4 Cross-Language Type Inference
Dynamically-typed languages (Python, JavaScript) require type inference to populate the IR.
Type inference is inherently incomplete for fully dynamic code.

Mitigation: Unresolvable types fall back to a Dynamic IR type. The code is still compressed
structurally; only the type-specific compression is lost for those identifiers. Partial type inference
(from type hints, usage patterns, and return value construction) captures the majority of types in
practice.

17.5 Round-Trip Fidelity for Formatting
The compression discards whitespace and comments. Exact byte-level round-trip requires
preserving these in a sidecar file.

Mitigation: For the AI consumption use case, formatting and comments are irrelevant. The
expanded output is reformatted by the language's standard formatter (rustfmt, black, prettier).
Comments from the original source are reattached by position from the sidecar file if exact
preservation is required.

18. Related Work
Code minification [8, 9] reduces JavaScript file size for network transfer. Minification shares the
variable-renaming approach but targets byte count rather than token count, does not consider
tokenizer behavior, and provides no bidirectional mapping.

Retrieval-augmented generation [1] optimizes which code to include in context. Our approach
optimizes how that code is represented. The two approaches are complementary and compose
multiplicatively.

Repository mapping [2] summarizes codebase structure for context inclusion. This is a lossy
compression that discards code in favor of summaries. Our approach is lossless.

Byte Pair Encoding [6] is the tokenization algorithm used by most language models. Our custom
vocabulary layer extends BPE with domain-specific merges for code patterns.

Huffman coding [11] assigns shorter codes to more frequent symbols. Our frequency-optimized
identifier assignment applies this principle at the identifier level.

Shannon information theory [7] provides the theoretical lower bound on compression. Our
analysis in Section 14 uses Shannon entropy to establish the theoretical minimum and measure
compression efficiency.

Tree-sitter [12] provides incremental parsing for multiple programming languages. We use
tree-sitter as the parsing frontend for non-Rust language adapters.

LoRA fine-tuning [13] enables efficient model adaptation. While not directly part of the
compression system, LoRA enables the MO prediction model to be fine-tuned on
developer-specific patterns with minimal compute.

19. Future Work
19.1 Empirical Validation
The compression ratios and model performance projections in this paper are derived from
analysis and small-scale testing. Large-scale empirical validation across multiple codebases,
languages, models, and developers is needed to confirm the projected benefits.

19.2 Tokenizer Co-Optimization
Training a tokenizer specifically on compact code (rather than adding vocabulary to an existing
tokenizer) could yield additional compression by optimizing the entire vocabulary for the
compact representation.

19.3 Model Fine-Tuning on Compact Code
Fine-tuning a language model on compact code input/output pairs could improve the model's
comprehension of the compact representation, potentially closing the gap between compact and
human-readable input quality.
19.4 Arithmetic Coding
Replacing the symbolic dictionary with arithmetic coding [14] could approach the Shannon
entropy limit more closely, though at the cost of human-inspectability of the compressed
representation.

19.5 Context-Dependent Symbol Resolution
Allowing the same symbol to carry different meanings in different positions (as in natural
language) could reduce the dictionary size while maintaining compression ratio. This requires
the model to perform contextual disambiguation, which current models handle well for natural
language but has not been tested for code representations.

19.6 Integration with Retrieval Systems
Combining compression with intelligent retrieval (RAG) would optimize both which code to
include and how to represent it. The combined system would select the most relevant code
fragments and present them in maximally compressed form, achieving the highest possible
information density within the context window.

20. Conclusion
The context window is the fundamental bottleneck of AI-assisted code generation. Current
approaches optimize which code to show the model. This paper demonstrates that optimizing
how code is represented yields 3.2x improvement through mechanical transformation alone,
extending to 22x with personalization — approaching the Shannon entropy limit of the source
code's information content.

The key insight is that human-readable source code is not optimized for language model
consumption. Descriptive identifiers, formatting, verbose error messages, and repeated
structural patterns all consume tokens that carry information for humans but not for models. By
maintaining two representations — one for humans, one for models — and converting losslessly
between them, we can dramatically increase the amount of code a model can process within its
context window.

The practical implication is significant: a 3-billion parameter model running on consumer
hardware with 3.2x compression sees more code than the same model without compression.
With full personalization at 22x compression, it sees more code than a 70B model without
compression. This does not make the small model reason better, but for the large class of tasks
where the bottleneck is context coverage rather than reasoning capability, compression directly
translates to improved output quality.

The system requires no model modification, no cloud infrastructure, and no recurring cost. Layer
1 compression is implementable in one week and provides immediate measurable benefit. Each
subsequent layer adds compression incrementally, with personalization layers improving
automatically over time as developer interaction data accumulates.
We have not made the model bigger. We have made the code smaller. The result is the same:
the model sees more, understands more, and generates better code.

21. References
[1] Lewis, P., et al. "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks."
NeurIPS 2020.

[2] Gauthier, P. "Aider: AI Pair Programming in Your Terminal." , 2023.

[3] Continue.dev. "Context Providers for AI-Assisted Development." , 2023.

[4] OpenAI. "GPT-4 Technical Report." arXiv:2303.08774, 2023.

[5] Liu, N., et al. "Lost in the Middle: How Language Models Use Long Contexts." TACL, 2024.

[6] Sennrich, R., et al. "Neural Machine Translation of Rare Words with Subword Units." ACL
2016.

[7] Shannon, C.E. "A Mathematical Theory of Communication." Bell System Technical Journal,
1948.

[8] Terser. JavaScript Mangler and Compressor.

[9] UglifyJS. JavaScript Parser, Minifier, Compressor.

[10] ProGuard. Java Class File Shrinker and Obfuscator.

[11] Huffman, D.A. "A Method for the Construction of Minimum-Redundancy Codes."
Proceedings of the IRE, 1952.

[12] Brunsfeld, M. "Tree-sitter: An Incremental Parsing System for Programming Tools."

[13] Hu, E.J., et al. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR 2022.

[14] Witten, I.H., et al. "Arithmetic Coding for Data Compression." Communications of the ACM,
1987.

Copyright 2024 Michael Cochran. All rights reserved.
