Compiling Windows Event Log templates

evtx renders event logs about three times faster than it did two releases ago. The change: each event template is now compiled once into pre-rendered output text plus a short list of fill-in-the-blank instructions, instead of being re-walked for every record.

June 2026

The shape of the problem

An EVTX file is a sequence of 64 KiB chunks, each holding a few hundred records. A record's payload is binary XML: a tokenized tree format in which Microsoft made one genuinely good decision — structure and data are stored separately. The structure lives in a template definition (an element tree with typed holes), stored once per chunk. Each record is then just a reference to a template plus an array of typed values for the holes.

Rendered, a record looks like this — a Security 4688, process creation:

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-…}"/>
    <EventID>4688</EventID>
    …
    <TimeCreated SystemTime="2016-10-06T01:47:07.166302Z"/>
    <Computer>WIN-WFBHIBE5GXZ.example.co.jp</Computer>
  </System>
  <EventData>
    <Data Name="NewProcessName">C:\Windows\System32\cmd.exe</Data>
    …
  </EventData>
</Event>

Almost every byte of that output is constant per template — only the values change between records. And the workload is extremely repetitive: the 30 MB benchmark log holds 62,000 records across 481 chunks but only 32 distinct templates. Until this release, the parser re-parsed each template's structure once per chunk (about 4,500 times per file) and re-walked it once per record.

What each version did per record

A profile of v0.12 showed where the time went: roughly a quarter of it decoded values into an intermediate enum, a sixth dispatched the tree walk, a sixth escaped and wrote strings, and a seventh ran layout classification — per record, for structure that is identical across thousands of records. All of that is computable once per template.

Compiling a template

The idea is the same as a printf format string, prepared ahead of time. The compiler walks a template definition once and renders everything constant — tag names, attributes, indentation, escaping — into a single flat text buffer. What's left is a short list of instructions: copy this range of the buffer, format value N here. Per record, rendering is just running that list.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Microsoft-Windows-Security-Auditing" Guid="{54849625-…}"/>
    <EventID>⟨value 3 · UInt16⟩</EventID>
    <TimeCreated SystemTime="⟨value 7 · FileTime⟩"/>
    <EventRecordID>⟨value 10 · UInt64⟩</EventRecordID>
    <Execution ProcessID="⟨value 8 · UInt32⟩" ThreadID="⟨value 9 · UInt32⟩"/>
    <Computer>⟨value 12 · UTF-16 string⟩</Computer>
  </System>
  ⟨value 19 · embedded binary XML — the EventData section⟩
</Event>

In the source, the XML instruction set is one five-variant enum. Each variant carries everything its runtime decision needs, precomputed — including the alternative closing sequences for an element whose content might be empty, text, or a nested element:

enum XOp {
    /// Copy lits[range] to the output.
    Lit(LitRange),
    /// Escaped value text (element content or an always-present attribute).
    Val { slot: u16, in_attr: bool },
    /// ` name="value"` — the whole attribute is omitted when the value is empty.
    AttrVal { slot: u16, pre: LitRange },
    /// Element content with one hole: empty / text / nested-element each
    /// get a precompiled closing sequence.
    Body { slot: u16, optional: bool, indent: u16,
           tail_text: LitRange, tail_empty: LitRange, tail_elem: LitRange },
    /// A hole in child position: nothing, a nested instance, or indented text.
    ChildSlot { slot: u16, optional: bool, indent: u16, ind: LitRange },
}

JSON compiles from the same template with its own instruction set (Lit, LeafVal, Elem, SlotChild) because JSON output has its own rules, and they're decided at compile time too: EventData flattening, duplicate-key _N suffixes, which wire types print as bare numbers (integers and booleans; floats, hex, timestamps, GUIDs and SIDs stay quoted strings), and null versus "" for empty values.

Running one record through it

At render time, a record contributes a value table: a count, then one (size, type) descriptor per value, then the value bytes packed back to back. A validation pass turns the descriptors into typed windows into the chunk buffer — no decoding, no copying:

Then the instructions run:

And the output is assembled — gray bytes come straight out of the buffer with memcpy, colored bytes are formatted from the record:

Each value is formatted exactly once, directly from chunk bytes into the output buffer — UTF-16 to UTF-8 with escaping fused in a SIMD pass, integers through fixed-width decoders. Values never materialize into an intermediate representation at all. Nested templates (value 19 above) are themselves compiled, so the EventData section runs the same way.

The validation pass in front of this is what keeps it safe. Descriptor sizes are checked against a 256-entry table of wire-format facts:

const TY_CLASS: [TyClass; 256] = { /* NULL→Sized, BOOL→Fixed(4),
    GUID→Fixed(16), SID→Sid, UTF16_STRING→Utf16, …, everything else→Reject */ };

A size that doesn't match its type, an unknown type, a malformed nested instance — anything irregular — and the record is rejected before a single byte of output is written. Rejected records take the slow path.

The slow path

Some records can't run on a compiled template: the compiler turns down templates whose output structure depends on values in ways the instruction set doesn't express, the per-record validation rejects irregular value tables, and the --separate-json-attributes output mode isn't compiled at all. Those records are parsed into a full tree and rendered from that, the way v0.12 rendered everything.

The part worth being careful about: the slow path is not a second rendering implementation. The compiler type itself renders trees. Walking a template, it emits an instruction wherever it finds a hole; walking a fully parsed record tree, there are no holes left, and the same walk writes its text directly to the output instead. Layout classification, escaping, emptiness rules — one copy of each. This matters because the path is chosen per record, and a record must render to the same bytes no matter which path it took.

One case deserved real instructions instead of the slow path: array values. A substitution can hold a string array, and the spec says the containing element repeats once per item — <Data>a</Data><Data>b</Data> from a single two-item value. System and Application logs have these in 17–23% of records. They get a dedicated instruction that owns its element's opening tag and replays it per item; on a Zen 2 core that took Application.evtx from 22.8 ms to 11.0 ms.

Compile once per file, not per chunk

Compiled templates own their bytes, with no lifetime ties to the chunk they came from, so they can be cached at two levels: a per-chunk map keyed by definition offset for the common case, backed by a parser-wide store keyed by template content — GUID, size, and a hash of the definition bytes — behind an RwLock and shared across worker threads via Arc. The 481 chunks of the benchmark file used to trigger ~4,500 template parses per read; now each of the 32 templates compiles exactly once per run, and multithreaded workers reuse each other's work.

Results

security_big_sample.evtx — 30 MB, 62k records — single-threaded (-t 1), hyperfine, 10 runs. Three releases on the same Apple M3 Pro:

On an AMD Zen 2 box pinned to one core — the machine every change was measured on before landing — the gap against 0.12.1 is wider:

Verification

• 27 sample logs covering the corpus's odd shapes — ANSI codepages, provider-cached templates, zero-record files — are rendered in all five output modes and hashed against the previous release after every commit. This caught one real bug during development (generic binary-XML fragments in JSON output) before it landed.
• 99 integration tests plus snapshot tests, which also cover the per-record APIs.
• A review pass over the new code found one genuine pre-existing bug — unbounded recursion on crafted nesting chains — now bounded by depth caps at build, validation, and scan time.
• The crate is #![forbid(unsafe_code)] throughout, including the SIMD UTF-16 conversion.

Benchmarks: hyperfine -w2 -r10, evtx_dump -t 1 -o <fmt> to /dev/null, fast-alloc feature, Apple M3 Pro (macOS) and AMD Zen 2 (Linux, taskset-pinned). The sample logs and output hashes are reproducible from the repository.