Architecture Deep Dive

How TidyView Works

Virtual views over columnar data. No copies until you ask for them.
Every filter flips bits — not rows. Every chain shares the same base DataFrame.

TidyView = Rc<DataFrame> + BitMask + ProjectionMap + Option<Ordering>

Packed u64 Bitmask

One bit per row, packed into 64-bit words. Filters AND their bitmasks together. No row data is ever copied during filtering.

1M rows = ~122 KB

Column Projection

select() narrows an index of visible column positions. Column data stays untouched — only the map of which columns you see changes.

O(ncols) memory

Lazy Sort

arrange() stores a sort permutation without reordering data. Only when you call materialize() does the concrete sorted DataFrame get built.

Zero-copy until needed
Lazy Operation Chain (all share the same Rc<DataFrame>)
filter()
Flip bits in mask
→
select()
Narrow projection
→
group_by()
BTreeMap groups
→
summarise()
Kahan-compensated
→
materialize()
Allocates here only
⚡

Filters Flip Bits, Not Rows

Columnar predicates scan typed vectors word-at-a-time. Chained filters AND their bitmasks — no intermediate DataFrames created.

🔄

Rc<DataFrame> Shared Base

Every view in a chain holds a reference-counted pointer to the same base data. Memory grows by the bitmask size, not the data size.

✅

Kahan Summation Everywhere

Sum, mean, variance, standard deviation — all use compensated accumulation. No floating-point drift even over millions of values.

🔒

BTreeMap, Never HashMap

Every grouping, every join index uses BTreeMap/BTreeSet for deterministic iteration order. Same input = bit-identical output.

Performance Benefits
Dict Filter (flat)
440 µs
Dict Filter (encoded)
98 µs — 4.5× faster
Expression gap (before)
4.7× overhead
Expression gap (after)
1.71× — 63% reduction
Join (BTreeMap)
29× slower for sorted keys
Join (SortMerge)
29× faster
Before optimization
TidyView optimized