{"id":"lessence-1kx","title":"Threshold sweep: optimal default from 85 to 75","description":"Swept --threshold values across 16 example logs to find optimal default. The original 85 was a magic number with no empirical basis.","design":"## Methodology\n\nSwept 9 threshold values (50, 60, 70, 75, 80, 85, 90, 95, 100) across 16\nexample log files, then narrowed to 72-78 range with quality spot-checks.\n\n### Test corpus (16 files)\napache (100), apiserver (4.8k), argocd-controller (28k), argocd-server (61k),\ncilium (38k), harbor-postgres-primary (54k), harbor-postgres-replica (24k),\nhdfs (100), kubelet (71k), microservices (53), nginx (50), prometheus (100),\nrancher (22k), realistic-repetitive (70), security-attacks (64), vllm (3.2k)\n\n## Results\n\n### Wide sweep (avg across 16 files)\n| Threshold | Avg Compression | Avg Output Lines |\n|-----------|----------------|-----------------|\n| 50% | 88.5% | 52 |\n| 60% | 87.6% | 87 |\n| 70% | 82.0% | 178 |\n| 75% | 80.3% | 197 |\n| 80% | 79.9% | 215 |\n| 85% | 77.8% | 232 |\n| 90% | 75.0% | 252 |\n| 100% | 71.5% | 351 |\n\n### Narrow sweep (72-78)\n| Threshold | Avg Compression | Avg Output |\n|-----------|----------------|------------|\n| 72% | 82.0% | 181 |\n| 73% | 82.0% | 185 |\n| 74% | 80.8% | 191 |\n| 75% | 80.3% | 197 |\n| 76% | 80.1% | 201 |\n| 77% | 79.9% | 206 |\n| 78% | 80.0% | 209 |\n\n## Quality cliff at 73\n\nAt threshold=73, nginx log merges HTTP 304 and 200 responses into one group.\nAt 74+, they correctly separate. This is the critical boundary — below 74,\ndifferent HTTP status codes (different semantic meaning) get merged.\n\nSecurity log at 73 merges SSH admin+root attempts. At 74+ they stay separate.\n\n## Key findings by log type\n\n- **HDFS**: 85 splits identical \"delete block\" ops into 5 groups (by block ID).\n  75 correctly merges to 1 group.\n- **Prometheus**: Same pattern — \"Deleting obsolete block\" splits at 85, merges at 75.\n- **Security**: All attack types stay properly separated at 75. At 50-60, SSH/SQL/rate-limit bleed together.\n- **k8s (kubelet, cilium, rancher)**: 1-2% better compression at 75 with no quality loss.\n- **nginx**: Correctly separates status codes at 74+, merges them at 73-.\n\n## Decision\nChanged default from 85 → 75. The 85 was a magic number from the first commit\nwith no testing. 75 gives 3% better avg compression with zero quality regressions.","status":"closed","priority":3,"issue_type":"research","created_at":"2026-04-03T02:43:33.783944901Z","created_by":"claude","updated_at":"2026-04-03T02:44:10.459710264Z","closed_at":"2026-04-03T02:44:10.459458435Z","close_reason":"Research complete. Default changed 85→75 in commit b80432c.","source_repo":".","compaction_level":0,"original_size":0,"labels":["autoresearch","perf"]}
{"id":"lessence-2k4","title":"Remove --max-tokens flag","description":"Remove the --max-tokens flag entirely. The fake token counting (words/0.75) is dishonest\nand the use case (cap output for LLM context windows) is better served by head -n or\na future --max-output-lines flag.\n\n1. Remove max_tokens field from Cli struct in src/main.rs\n2. Remove parse_token_count function from src/main.rs\n3. Remove max_tokens from Config struct in src/config.rs\n4. Remove all token counting/limiting logic in folder.rs and main.rs\n5. Remove count_tokens methods from folder.rs and analyzer.rs\n6. Update any tests that reference max_tokens\n7. Remove from CLAUDE.md flags list\n\nAcceptance: --max-tokens is not in lessence --help and cargo test passes.","status":"closed","priority":2,"issue_type":"task","created_at":"2026-04-01T13:51:43.950035921Z","created_by":"jorgen","updated_at":"2026-04-01T14:51:46.460436799Z","closed_at":"2026-04-01T14:51:46.460266145Z","close_reason":"Removed --max-tokens flag and all fake token counting","source_repo":".","compaction_level":0,"original_size":0,"labels":["cleanup","cli"]}
{"id":"lessence-6mq","title":"Unit tests: remaining pattern detectors (quoted, process, email, uuid)","description":"Add unit tests for remaining pattern detector functions.\n\n1. QuotedStringDetector::detect_and_replace (20 mutants): test single/double quotes, nested quotes, empty strings.\n2. ProcessDetector::detect_and_replace (20 mutants): test PID, thread ID detection.\n3. EmailDetector functions (various): test valid emails, false positives, edge cases.\n4. UuidDetector functions (various): test UUID v4, v5, request IDs, trace IDs.\n5. HashDetector functions (various): test MD5, SHA1, SHA256, SHA512 detection.\n6. HttpStatusDetector functions (various): test 2xx/3xx/4xx/5xx, false positives.\n\nLower priority because these have some existing unit tests already (tests/unit/).","status":"open","priority":3,"issue_type":"task","created_at":"2026-04-12T12:22:16.499085779Z","created_by":"jorgen","updated_at":"2026-04-12T12:22:16.499085779Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["mutants","patterns","testing"]}
{"id":"lessence-7jb","title":"Create agent skill for lessence","description":"Create .claude/skills/lessence-use/SKILL.md teaching coding agents when/how to use lessence.\n\n1. Same pattern as playwright-cli skill: judgment, not documentation\n2. When to use: test failure 500+ lines, kubectl dumps, CI failures, repetitive walls\n3. When NOT to: structured JSON (jq), specific known error (grep), small output (<50 lines)\n4. Interpret output: [+N similar] = repeating problem, many groups = diverse problems\n5. Ships with current feature set, improves as new features land\n6. Example invocations for common scenarios (k8s, CI, crash loops)\n\nAcceptance: skill file loads in Claude Code and triggers on relevant contexts.","status":"closed","priority":2,"issue_type":"feature","created_at":"2026-04-01T12:31:46.138956501Z","created_by":"jorgen","updated_at":"2026-04-01T22:45:57.901668192Z","closed_at":"2026-04-01T22:45:57.901494194Z","close_reason":"Skill created at ~/.claude/skills/lessence/","source_repo":".","compaction_level":0,"original_size":0,"labels":["agent","skill"],"comments":[{"id":1,"issue_id":"lessence-7jb","author":"claude","text":"Created as user-profile skill at ~/.claude/skills/lessence/SKILL.md instead of project-scoped. Triggers on verbose log scenarios across all projects.","created_at":"2026-04-01T22:45:57Z"}]}
{"id":"lessence-cft","title":"Top-N frequency mode (--top N)","description":"Show only the N most frequent patterns, sorted by count descending.\n\n1. New flag: --top N implies sort by frequency desc (no separate --sort)\n2. Output: count prominently shown, e.g. \"[1247x] Connection refused from <IP>\"\n3. Footer: \"(showing top 5 of 142 patterns, 98.2% of lines)\"\n4. Data already available from folding (group counts), needs reorder + truncate\n\nPrior art: logmine sorts by cluster size by default. angle-grinder has\ncomposable pipeline. --top N is the simplest UX for a pipe tool.\n\nAcceptance: lessence --top 5 shows exactly 5 groups sorted by count desc.","status":"closed","priority":1,"issue_type":"feature","created_at":"2026-04-01T12:31:22.765305027Z","created_by":"jorgen","updated_at":"2026-04-01T14:00:51.302628645Z","closed_at":"2026-04-01T14:00:51.302457401Z","close_reason":"Implemented --top N flag with frequency-sorted output","source_repo":".","compaction_level":0,"original_size":0,"labels":["cli","triage"]}
{"id":"lessence-d17","title":"Synthetic log generator for constitutional tests","description":"Completed: tests/fixtures/log_generator.rs generates deterministic\nkubelet-style logs with configurable line count. Used by\ntest_constitutional_compliance_generated which runs in CI without\ncorpus files.\n\n1. Generator covers: UUID, IP, Hash, Path, PodName, Namespace, PID, Timestamp, Duration, QuotedString\n2. 1000 lines = 98.5% compression, 500 lines = 97%\n3. Runs in debug and release builds\n4. Kubelet.log tests kept as bonus local-only coverage","status":"closed","priority":3,"issue_type":"task","created_at":"2026-04-12T13:13:37.020473809Z","created_by":"jorgen","updated_at":"2026-04-12T13:13:37.095808143Z","closed_at":"2026-04-12T13:13:37.095602240Z","close_reason":"Implemented in commit 91895ee","source_repo":".","compaction_level":0,"original_size":0,"labels":["generator","testing"]}
{"id":"lessence-dno","title":"Accept file arguments","description":"Add optional positional file args to clap, fall back to stdin when none given.\n\n1. Add files: Vec<PathBuf> positional arg to Cli struct\n2. Support - as explicit stdin (universal convention: cat, rg, bat, jq)\n3. Multiple files concatenated into one stream (like cat/jq)\n4. Shell handles glob expansion (no built-in glob needed)\n5. Error handling: warn and skip bad files, process the rest (like cat)\n6. Binary detection: check first 8KB for null bytes, warn and skip\n\nAcceptance: lessence app.log produces same output as lessence < app.log.\nItems 1 and 5 in .ideas/features.md are duplicates of this.","status":"closed","priority":1,"issue_type":"feature","created_at":"2026-04-01T12:31:07.834205110Z","created_by":"jorgen","updated_at":"2026-04-01T13:11:35.953423242Z","closed_at":"2026-04-01T13:11:35.953252070Z","close_reason":"Implemented file argument support","source_repo":".","compaction_level":0,"original_size":0,"labels":["cli","ux"]}
{"id":"lessence-dui","title":"Unit tests: normalize.rs similarity_score","description":"Add unit tests for Normalizer::similarity_score in src/normalize.rs to kill remaining mutants.\n\n1. similarity_score (32 mutants from quick run, 30 caught, 2 missed): math operators in Levenshtein distance calculation. Pin exact scores for known input pairs. Test symmetric property, self-similarity=100, empty strings, identical strings, completely different strings.\n2. normalize_line (8 mutants from quick run): test short-circuit logic, OR operator mutations. Test empty input, single token, multiple tokens.\n\nProptest already covers commutativity and self-similarity, but unit tests with pinned values would catch operator mutations.","status":"open","priority":3,"issue_type":"task","created_at":"2026-04-12T12:22:06.898333111Z","created_by":"jorgen","updated_at":"2026-04-12T12:22:06.898333111Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["mutants","normalize","testing"]}
{"id":"lessence-ej1","title":"Clean up help text (remove LLM references)","description":"Remove LLM positioning from CLI help descriptions.\n\n4 references in src/main.rs:\n1. Line 17: strip_ansi_codes doc \"for LLM consumption\" -> remove\n2. Line 98: Preflight \"for LLM agents\" -> \"for automation/CI\"\n3. Line 122: preserve_color \"for LLM optimization\" -> remove qualifier\n4. Line 126: max_tokens \"for LLM limits\" -> \"limit output size\"\n\nAcceptance: lessence --help contains zero mentions of LLM.","status":"closed","priority":1,"issue_type":"task","created_at":"2026-04-01T12:31:12.377253276Z","created_by":"jorgen","updated_at":"2026-04-01T13:15:40.973581117Z","closed_at":"2026-04-01T13:15:40.973395452Z","close_reason":"Removed all 4 LLM references from help text","source_repo":".","compaction_level":0,"original_size":0,"labels":["branding","cli"]}
{"id":"lessence-h08","title":"Unit tests: pattern detectors (timestamp, network, path)","description":"Add unit tests for pattern detector functions that have high mutant counts.\n\n1. UnifiedTimestampDetector::has_timestamp_indicators (66 mutants): checks if a string has timestamp-like patterns. Test positive/negative cases.\n2. NetworkDetector::detect_and_replace (58 mutants): detects IPs, ports, FQDNs. Test IPv4, IPv6, port ranges, false positives.\n3. NetworkDetector::is_plausible_ipv (26 mutants): validates IP-like strings. Test boundary cases.\n4. PathDetector::is_likely_file_path (30 mutants): classifies file paths. Test Unix, Windows, relative, false positives.\n5. PathDetector::is_likely_url_path (28 mutants): classifies URL paths. Test http/https, false positives.\n6. DurationDetector::detect_and_replace (38 mutants): detects time durations. Test ms, s, m, h formats.\n\nTests go in each pattern module's cfg(test) section or tests/unit/.","status":"open","priority":2,"issue_type":"task","created_at":"2026-04-12T12:21:40.854283642Z","created_by":"jorgen","updated_at":"2026-04-12T12:21:40.854283642Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["mutants","patterns","testing"]}
{"id":"lessence-htq","title":"Investigate cargo-mutants memory blowup","description":"cargo test alone uses 187MB peak. cargo-mutants process uses 430MB.\nBut when mutants runs cargo test inside a systemd scope, system memory\nclimbs to 44GB+ and triggers OOM kills.\n\nKnown facts:\n1. Normal cargo test: 187MB peak, 16s wall time, 335 tests pass\n2. cargo-mutants process: 430MB peak (from /usr/bin/time -v)\n3. Inside systemd scope: OOM killed at 14.7GB RSS on the test binary\n4. --gitignore was false by default, copying 550MB of fuzz artifacts (now fixed)\n5. With --gitignore true, copy is ~10MB but memory still blows up\n6. 1 job, not parallel\n7. Chrome/VS Code use ~10GB normally, system has 62GB total\n\nUnanswered: what inside cargo-mutants' execution causes 14.7GB RSS\non the test binary when the same binary uses 187MB standalone?\nPossible leads: mutants' temp dir build cache, incremental compilation\nartifacts, multiple test binaries compiled simultaneously, or\nsomething about how cargo test behaves differently in the temp tree.\n\nReproduce: make mutants (observe memory via htop or /usr/bin/time -v)","status":"open","priority":1,"issue_type":"bug","created_at":"2026-04-12T14:21:06.220849069Z","created_by":"jorgen","updated_at":"2026-04-12T14:21:06.220849069Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["memory","mutants","testing"]}
{"id":"lessence-i78","title":"Unit tests: stats reporting functions","description":"Add unit tests for stats reporting functions in src/folder.rs to kill mutants.\n\n1. count_pattern_types (84 mutants): increments stats counters per Token variant. Test each token type maps to correct counter. Tricky: Port increments ips, Json increments paths, BracketContext/KV/etc increment percentages.\n2. count_active_pattern_types (110 mutants): counts non-zero stats fields. Test 0/1/all non-zero. Note: emails field may be unchecked (potential bug).\n3. print_stats (104 mutants): writes markdown report to writer. Test with Vec<u8>, assert sections present, pattern rows only when count > 0, compression ratio thresholds.\n4. print_stats_json (mutants): JSON to stderr. Test valid JSON, all fields present.\n5. print_summary_json (mutants): JSON with type=summary. Test flattened structure, all fields.\n\nAll private functions, tests go in cfg(test) mod tests in folder.rs.","status":"open","priority":2,"issue_type":"task","created_at":"2026-04-12T12:21:03.060783048Z","created_by":"jorgen","updated_at":"2026-04-12T12:21:03.060783048Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["folder","mutants","testing"]}
{"id":"lessence-il5","title":"Unit test: format_group_json stats counting","description":"Missed mutant: src/folder.rs:759 replace && with || in format_group_json.\n\nThe condition gates whether collapsed_groups and lines_saved stats are\nincremented in JSON mode. Changing && to || goes undetected because no\ntest asserts on these stats fields in JSON output specifically.\n\nFix: add a unit test that processes lines through JSON mode and verifies\ncollapsed_groups and lines_saved match expected values. Test both with\nand without essence_mode to exercise both branches of the condition.\n\nPart of lessence-okn (format_group unit tests).","status":"open","priority":2,"issue_type":"task","created_at":"2026-04-12T14:39:33.937149881Z","created_by":"jorgen","updated_at":"2026-04-12T14:39:33.937149881Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["folder","mutants","testing"]}
{"id":"lessence-mp5","title":"Research: line truncation in CLI summary output","description":"Researched how CLI tools handle long lines in output. Findings inform whether lessence --summary should truncate.","design":"## Question\nShould lessence --summary truncate long representative lines? Currently\nhardcoded at 200 chars with \"...\". No research was done before picking 200.\n\n## How Other Tools Handle It\n\n### Search tools\n- **ripgrep**: No truncation by default. Opt-in via --max-columns N (omits line)\n  or --max-columns N + --max-columns-preview (truncates with prefix). Recommended\n  config in GUIDE.md: --max-columns=150.\n- **ag (silver searcher)**: --width NUM truncates. No default. Separately skips\n  lines >2k chars by default.\n- **bat**: Auto-wraps to terminal width by default. --chop-long-lines to truncate\n  instead of wrap. --terminal-width for explicit override.\n- **ack**: No truncation at all.\n\n### Log tools\n- **lnav**: Most sophisticated — per-field max_width with overflow algorithms:\n  truncate, dot-dot (middle ellipsis), abbrev (com.example → c.e), last-word.\n  Config-based, not CLI flag.\n- **angle-grinder**: Adapts to terminal width, no explicit flag.\n- **logmine, drain3**: No truncation.\n\n### System tools\n- **docker ps**: Hardcoded per-column: COMMAND=20 chars, ID=12 chars. --no-trunc\n  to disable. Does NOT detect terminal width.\n- **kubectl get**: No truncation at all. Infamously wide (k8s issue #7843).\n- **git log --oneline**: No truncation. Full first line regardless of length.\n- **ps**: Truncates COMMAND to terminal width. -ww for unlimited.\n\n### Terminal width detection\n- Tools use ioctl(TIOCGWINSZ) or tput cols. $COLUMNS is shell-local, not\n  inherited. Non-TTY (pipes) can't detect width — fall back to 80 or unlimited.\n\n## Findings\n1. No tool hardcodes 200 — arbitrary number with no precedent.\n2. The dominant pattern: no truncation by default, opt-in flag for users who want it.\n3. ripgrep's --max-columns is the most cited best practice for search/summary tools.\n4. Terminal-width-aware auto-truncation (bat style) is elegant for TTY, but\n   lessence output is often piped, where width detection fails.\n\n## Recommendation\nRemove the hardcoded 200-char truncation from --summary. Instead:\n- No truncation by default (matches ripgrep, git, kubectl)\n- Future: add --max-columns N flag if users request it\n- The agent that reported 103KB output should use --top N, not ask for truncation\n\n## Sources\n- ripgrep GUIDE.md: --max-columns=150 recommended config\n- ripgrep issue #522: discussion on defaulting max-columns\n- ag manpage: --width NUM flag\n- bat manpage: --wrap auto, --chop-long-lines, --terminal-width\n- lnav formats docs: max_width + overflow algorithms\n- docker/cli container.go source: Ellipsis(command, 20)\n- kubectl issue #7843: \"laughably wide\" output complaint\n- git-log docs: --oneline shows full first line","status":"closed","priority":3,"issue_type":"research","created_at":"2026-04-03T03:07:26.657734319Z","created_by":"claude","updated_at":"2026-04-03T03:07:51.048721145Z","closed_at":"2026-04-03T03:07:51.048469257Z","close_reason":"Research complete. Recommendation: remove hardcoded 200-char truncation, no truncation by default.","source_repo":".","compaction_level":0,"original_size":0,"labels":["research","ux"]}
{"id":"lessence-mxv","title":"Wire up --disable-patterns for all accepted names (fix silent no-ops)","description":"`--disable-patterns` has three conflicting sources of truth, causing silent\nno-ops for 8 of the 16 names it accepts today.\n\n## The inconsistency\n\n| Source | Location | Names |\n|---|---|---|\n| `--help` text | `main.rs:112` | 7 (missing `process`) |\n| `VALID_PATTERNS` validator | `main.rs:27-44` | 16 |\n| Actual wiring → `config.normalize_*` | `main.rs:218-229` | 8 |\n\nUser impact: `lessence --disable-patterns brackets` passes validation, prints\nnothing, does nothing. Fails silently — matches the \"must kill mutants, not\njust not-panic\" quality bar.\n\n## Current behavior breakdown\n\n**Tier A — honored (8):** timestamp, hash, network, uuid, process, email, path, duration.\n\n**Tier B — validator accepts, hardcoded-on, silent no-op (2):** json, kubernetes.\nSee `main.rs:227, 229` (`normalize_json: true, // Always enabled for modern\nlog formats` / `normalize_kubernetes: true, // Always enabled for cloud-native\nlogs`). Deliberate design — must either (a) remove from VALID_PATTERNS or\n(b) actually gate them. Author call.\n\n**Tier C — validator accepts, no config wiring exists, silent no-op (6):**\nhttp-status, brackets, key-value, quoted-string, name, decimal. These detectors\nrun unconditionally in `normalize.rs:115-175`.\n\n**Tier D — detectors that run but aren't even in VALID_PATTERNS (2):**\nlog_module, structured. `normalize.rs:141, 149` — no way to disable at all.\n\n## Work items\n\n1. **Add config fields** in `config.rs`: `normalize_http_status`, `normalize_brackets`,\n   `normalize_key_value`, `normalize_quoted`, `normalize_names`, `normalize_log_module`,\n   `normalize_structured`. Default all to `true`.\n\n2. **Wire the mapping** in `main.rs:218+`: map each validated name to its\n   `config.normalize_*` field. Rename `brackets` → `bracket_context`,\n   `quoted-string` → `quoted` in the VALID_PATTERNS list? (keep user-facing\n   names stable, map internally.) Also add `log-module` and `structured` to\n   VALID_PATTERNS.\n\n3. **Gate the detectors** in `normalize.rs`: wrap each `detect_and_replace`\n   call at lines 115, 123, 133, 141, 149, 167, 175 with a\n   `if self.config.normalize_X { ... }` guard. Mirror the pattern already\n   used for timestamps, paths, etc.\n\n4. **Decide json/kubernetes:** either (a) drop from VALID_PATTERNS and update\n   docs to say they're always-on by design; or (b) add real gating. Author\n   decision at implementation time — mention the trade-off in the commit.\n\n5. **Delete `decimal` from VALID_PATTERNS.** No `patterns/decimal.rs` exists.\n   Decimal numbers are detected inside `DurationDetector`\n   (`patterns/duration.rs:123-126`) — there is no meaningful way to extract\n   a separate decimal-only disable without changing tokenization. Users who\n   want to disable decimals already have `--disable-patterns duration`.\n   (This step is required regardless of Option A vs B.)\n\n6. **Sync the help text** at `main.rs:112`. It currently lists 7 names;\n   after this work it should list the real set.\n\n7. **Update docs:** `CLAUDE.md` (\"Valid pattern names (16)\") and the\n   `lessence` skill at `.claude/skills/lessence/references/flags.md` must\n   match the new set.\n\n8. **Add tests** per name — at least one test per disable flag that confirms\n   the detector's token type does NOT appear in the normalized output when\n   disabled. These are low-cost snapshot-style assertions that actually\n   exercise the wiring; cargo-mutants will catch regressions.\n\n## Out of scope\n\n- Refactoring normalize.rs detector dispatch into a table-driven loop. That's\n  a separate cleanup and would make step 3 cleaner, but it's a bigger change.\n- Exposing `--enable-only X,Y` as the inverse. Nice to have, not needed here.\n\n## Acceptance\n\n- `lessence --disable-patterns <any VALID_PATTERNS name>` either honors\n  the flag (token type disappears) or is rejected by the validator.\n- `--help` output, VALID_PATTERNS, and actual config wiring all list the\n  same names.\n- CLAUDE.md and lessence skill flags.md match.\n- `cargo test` passes with per-name disable assertions.","status":"closed","priority":1,"issue_type":"bug","created_at":"2026-04-15T22:39:12.873011093Z","created_by":"jorgen","updated_at":"2026-04-15T23:09:14.885935552Z","closed_at":"2026-04-15T23:09:14.884435966Z","close_reason":"Resolved by fe1ccf2 — all 7 silent-no-op names are now wired and guarded, with per-flag tests killing the 'forgot to wire this' mutant class. decimal handled separately in 8ef9666 (removed from VALID_PATTERNS, no backing code existed).","source_repo":".","compaction_level":0,"original_size":0,"labels":["cli","docs","patterns"],"comments":[{"id":2,"issue_id":"lessence-mxv","author":"jorgen","text":"`decimal` handled separately in 8ef9666 — removed from VALID_PATTERNS with a `fix:` commit that lands in release notes explaining the history (never implemented; use `duration` which owns all numeric tokens). Bead scope now covers only the remaining 7 silent no-ops: brackets, key-value, http-status, quoted-string, name (Tier C, need config wiring), plus json, kubernetes (Tier B, deliberately always-on — decide: drop from VALID_PATTERNS or add gating).","created_at":"2026-04-15T22:49:27Z"}]}
{"id":"lessence-nep","title":"Add -q/--quiet alias for --no-stats","description":"Add --quiet / -q as Unix-conventional alias for --no-stats.\n\n1. Add short alias -q and long alias --quiet to existing no_stats field\n2. Keep --no-stats working for backwards compatibility\n3. Standard Unix convention (grep -q, curl -q, rsync -q)\n\nScope: one line change in clap derive attributes.\nAcceptance: lessence -q suppresses stats footer.","status":"closed","priority":3,"issue_type":"feature","created_at":"2026-04-01T12:31:28.337505184Z","created_by":"jorgen","updated_at":"2026-04-01T14:55:31.036152894Z","closed_at":"2026-04-01T14:55:31.035975412Z","close_reason":"Added -q/--quiet alias for --no-stats","source_repo":".","compaction_level":0,"original_size":0,"labels":["cli","ux"]}
{"id":"lessence-okn","title":"Unit tests: format_group and output dispatch","description":"Add unit tests for group formatting functions in src/folder.rs.\n\n1. format_group (66 mutants): main text/markdown formatter. Takes PatternGroup + GroupRollup, returns formatted string. Test collapsed vs non-collapsed, essence mode, compact marker presence.\n2. format_group_json (22 mutants): JSON serialization of a group record. Test all fields present (id, count, normalized, token_types, first, last, time_range, variation). Test with/without timestamps.\n3. format_group_dispatch (2 mutants): routes to text or JSON based on config. Test both paths.\n4. finish_summary (76 mutants): renders summary mode output (one line per pattern). Test with various group counts, --top N interaction.\n5. finish_top_n (20 mutants): renders top N output sorted by count.\n\nBuild PatternGroups with make_group helper already in tests.","status":"open","priority":2,"issue_type":"task","created_at":"2026-04-12T12:21:14.747810769Z","created_by":"jorgen","updated_at":"2026-04-12T12:21:14.747810769Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["folder","mutants","testing"]}
{"id":"lessence-rc8","title":"Unit tests: pattern detectors (k8s, kv, structured, bracket)","description":"Add unit tests for newer pattern detector functions with high mutant counts.\n\n1. KeyValueDetector::classify_value_type (52 mutants): classifies KV pair value types. Test string, number, bool, path, duration.\n2. KeyValueDetector::has_key_value_indicators (30 mutants): checks for KV patterns. Test key=value, key:value, false positives.\n3. KeyValueDetector::is_valid_key_value_context (28 mutants): validates KV context.\n4. KeyValueDetector::is_common_config_pattern (24 mutants): identifies config patterns.\n5. StructuredMessageDetector::has_kubernetes_indicators (48 mutants): checks for k8s patterns.\n6. StructuredMessageDetector::has_structured_indicators (30 mutants): checks structured log indicators.\n7. StructuredMessageDetector::is_valid_structured_log (34 mutants): validates structured log format.\n8. StructuredMessageDetector::is_microservice_component (26 mutants): identifies microservice components.\n9. LogWithModuleDetector::has_kubernetes_indicators (36 mutants): k8s module detection.\n10. LogWithModuleDetector::has_log_module_indicators (34 mutants): log module detection.\n11. BracketContextDetector::has_kubernetes_indicators (36 mutants): k8s bracket detection.\n12. BracketContextDetector::has_bracket_indicators (30 mutants): bracket context detection.\n\nTests go in each pattern module's cfg(test) section.","status":"open","priority":2,"issue_type":"task","created_at":"2026-04-12T12:21:54.237012078Z","created_by":"jorgen","updated_at":"2026-04-12T12:21:54.237012078Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["mutants","patterns","testing"]}
{"id":"lessence-rfb","title":"JSON stats on stderr (--stats-json)","description":"Emit structured JSON statistics on stderr while compressed output goes to stdout.\n\n1. New flag: --stats-json emits JSON to stderr after stdout is flushed\n2. Fields: input_lines, output_lines, compression_ratio, groups, elapsed_ms, pattern_hits\n3. Orthogonal to --quiet: --quiet --stats-json gives clean stdout + JSON stderr\n4. Single JSON object at end (not NDJSON streaming)\n5. Use serde_json::to_writer(stderr.lock(), &stats)\n\nPrior art: ripgrep --stats prints to stderr. pv writes throughput to stderr.\nFuture: TTY auto-detection could auto-switch format (std::io::IsTerminal, Rust 1.70+).\n\nAcceptance: lessence --stats-json emits valid JSON on stderr with correct counts.","status":"closed","priority":2,"issue_type":"feature","created_at":"2026-04-01T12:31:24.663340914Z","created_by":"jorgen","updated_at":"2026-04-01T14:40:16.689120226Z","closed_at":"2026-04-01T14:40:16.688952674Z","close_reason":"Implemented --stats-json flag","source_repo":".","compaction_level":0,"original_size":0,"labels":["agent","cli"]}
{"id":"lessence-t6m","title":"Unit tests: pipeline and buffer management","description":"Add unit tests for pipeline functions in src/folder.rs.\n\n1. apply_pii_masking (12 mutants): public function, replaces email tokens with masked versions. Test with/without emails, mixed tokens, empty tokens.\n2. first_timestamp_in (8 mutants): extracts first Token::Timestamp from a token slice. Test with no timestamps, one timestamp, multiple timestamps, non-timestamp tokens only.\n3. flush_oldest_safe_group (24 mutants): removes oldest group from buffer, formats it. Test buffer state before/after, returned string.\n4. should_flush_buffer (6 mutants): returns true when buffer exceeds 1000 groups. Test at 999, 1000, 1001.\n5. apply_second_similarity_pass (54 mutants): re-clusters buffer groups. Test that similar groups merge.\n6. process_line (14 mutants): main hot path. Test single line processing, buffer growth.\n\nPrivate functions, tests in cfg(test) mod tests.","status":"open","priority":2,"issue_type":"task","created_at":"2026-04-12T12:21:28.067893931Z","created_by":"jorgen","updated_at":"2026-04-12T12:21:28.067893931Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["folder","mutants","testing"]}
{"id":"lessence-tc8","title":"Shell completions (--completions)","description":"Add shell completion generation using clap_complete.\n\n1. Add clap_complete = \"4\" to dependencies\n2. Add --completions <shell> flag (keeps lessence subcommand-free)\n3. Supports bash, zsh, fish, elvish, powershell via Shell enum\n4. Runtime generation (no build.rs needed)\n5. Usage: lessence --completions bash > ~/.local/share/bash-completion/completions/lessence\n\nScope: ~15 lines of code + one dependency.\nAcceptance: lessence --completions bash produces valid bash completion script.","status":"closed","priority":3,"issue_type":"feature","created_at":"2026-04-01T12:31:31.876561322Z","created_by":"jorgen","updated_at":"2026-04-01T15:04:38.383243027Z","closed_at":"2026-04-01T15:04:38.383061085Z","close_reason":"Added --completions flag via clap_complete","source_repo":".","compaction_level":0,"original_size":0,"labels":["cli","polish"]}
{"id":"lessence-tta","title":"Performance profiling: grouping pipeline optimization","description":"1. Profiled lessence pipeline on 50k synthetic log lines using hyperfine\n2. Identified 3 slowest components via isolation benchmarks\n3. Ran 14-iteration autoresearch optimize loop, 5 kept / 9 discarded\n4. Achieved 45% parallel speedup (1294ms to 712ms) and 54% worst-case improvement (8565ms to 3937ms)\n5. Net code reduction of 14 lines across folder.rs and normalize.rs","design":"## Bottom 3 Performance Bottlenecks (measured)\n\n### #1: Linear-scan grouping with Vec<char> allocation (FIXED)\nThe similarity_score() method allocated two Vec<char> per comparison call.\nWith 832 unique groups (worst case), that was 41.6M heap allocations across\n50k lines. O(n*k) where k = unique groups.\n\nFix: replaced with direct &[u8] byte comparison. Zero allocation.\nResult: worst-case 8565ms to 3880ms (55% faster).\n\n### #2: Non-disableable always-on detectors (PARTIALLY FIXED)\n7 pattern detectors run unconditionally on every line (HttpStatus, Brackets,\nKeyValue, LogModule, Structured, Name, QuotedString). Each allocates a String\neven when no pattern is found.\n\nFix: added cheap contains() pre-filters (skip detector if line has no '[',\n'=', '{', or quotes). Removed dead if-true guards.\nResult: ~100ms improvement on stress test. Full fix would require Cow<str>\nreturn type across all 17 detectors (future work).\n\n### #3: Rayon batch size too small (FIXED)\nDefault batch of 100 lines meant 500 rayon synchronization points per 50k\nlines. Each sync has overhead (thread pool wake, work stealing, barrier).\n\nFix: increased to 10,000 lines per batch (5 syncs per 50k lines).\nResult: parallel 1294ms to 712ms (45% faster).\n\n## Experiment Log\n\n| # | Hypothesis | Result | Status |\n|---|-----------|--------|--------|\n| 1 | HashMap for O(1) exact match | No gain, fuzzy path dominates | discarded |\n| 2 | Byte-level similarity (no Vec<char>) | 55% faster worst-case | KEPT |\n| 3 | Integer-only length rejection | Within noise | discarded |\n| 4 | HashMap + byte sim combined | Hash overhead eats gains | discarded |\n| 5 | Move-to-front heuristic | No gain on uniform data | discarded |\n| 6 | Vec pre-alloc + dedup branch | -10 lines, equal perf | KEPT (simpler) |\n| 7 | Lower flush threshold (1000 to 100) | Compression 74.8% to 56.7% | discarded (regression) |\n| 8 | Pre-filter always-on detectors | ~100ms gain, cleaner code | KEPT |\n| 9 | Skip initial String clone | Within noise, adds complexity | discarded |\n| 10 | Threshold-adaptive length rejection | Within noise | discarded |\n| 11 | Batch 100 to 1000 | Parallel 1294 to 808ms | KEPT |\n| 12 | Batch 1000 to 5000 | Parallel 808 to 738ms | KEPT |\n| 13 | Batch 5000 to 10000 | Parallel 738 to 712ms | KEPT |\n| 14 | Batch 10000 to 50000 | Plateau, worse memory | discarded |\n\n## Key Insight\nThe paradox: disabling all pattern detectors made the tool 2x SLOWER.\nBetter normalization = fewer unique groups = less similarity work.\nPerformance is dominated by the interaction between normalization quality\nand grouping cost, not by any single component.\n\n## Remaining Opportunities\n1. Cow<str> return type for detect_and_replace (avoid 800k String allocs)\n2. Hash-bucketed grouping (partition groups by normalized hash prefix)\n3. SIMD byte comparison in similarity_score (AVX2 on x86_64)\n4. Profile-guided optimization (PGO) for 10-20% overall improvement","status":"closed","priority":3,"issue_type":"research","created_at":"2026-04-01T21:13:22.277700422Z","created_by":"claude","updated_at":"2026-04-01T21:13:54.861695495Z","closed_at":"2026-04-01T21:13:54.861192943Z","close_reason":"Research complete, 4 optimizations merged to main via perf commit","source_repo":".","compaction_level":0,"original_size":0,"labels":["autoresearch","perf"]}
{"id":"lessence-w3k","title":"Guard apply_pii_masking loop against infinite loops","description":"src/folder.rs:37-42 — the while loop in apply_pii_masking advances\nstart by abs_pos + email.len(). If email is empty or the arithmetic\nis wrong, start never advances and the loop hangs forever.\n\nFound by cargo-mutants: replacing + with - on line 41 creates an\ninfinite loop that hangs for 69s until timeout kills it.\n\nFix: add two guards:\n1. Skip empty email tokens (continue)\n2. Check next_start > start before advancing (break if not)\n\nThis is a defensive code fix, not a test fix.","status":"open","priority":1,"issue_type":"bug","created_at":"2026-04-12T14:37:48.742343344Z","created_by":"jorgen","updated_at":"2026-04-12T14:37:48.742343344Z","source_repo":".","compaction_level":0,"original_size":0,"labels":["folder","safety"]}
{"id":"lessence-wjh","title":"CI exit codes (--fail-on-pattern)","description":"Exit non-zero when a regex pattern matches input lines. For CI pipeline gating.\n\n1. --fail-on-pattern \"ERROR|FATAL\" takes a regex\n2. Exit codes: 0=clean, 1=pattern matched, 2=tool error (trivy/hadolint convention)\n3. Always emit compressed output AND exit non-zero (diagnosis + gating)\n4. Scan raw lines during ingestion, set flag, check after output completes\n5. CI usage: lessence --fail-on-pattern ERROR < build.log || alert\n\nStart with regex only. Semantic --fail-on-level deferred until level awareness exists.\nAcceptance: exits 1 when input contains matching lines, 0 when it does not.","status":"closed","priority":3,"issue_type":"feature","created_at":"2026-04-01T12:31:38.021171188Z","created_by":"jorgen","updated_at":"2026-04-01T15:12:00.232673898Z","closed_at":"2026-04-01T15:12:00.232506008Z","close_reason":"Added --fail-on-pattern for CI exit code gating","source_repo":".","compaction_level":0,"original_size":0,"labels":["ci","cli"]}
