TIMBERFS(1)

NAME
timberfs - append-only, transparently compressed, write-time-indexed filesystem for log files

SYNOPSIS
timberfs mount [options] BACKING MOUNTPOINT
timberfs append [options] FILE
timberfs import [options] SOURCE... DEST
timberfs export [--from TIME] [--to TIME] SOURCE DEST
timberfs query [--from TIME] [--to TIME] FILE...
timberfs grep [options] PATTERN [FILE...]
timberfs index FILE
timberfs reindex FILE
timberfs rotate --cutoff TIME SOURCE [DEST] [--delete] [--dry-run]

DESCRIPTION
timberfs is a FUSE filesystem purpose-built for log files. Files in the mount look and behave like ordinary log files — loggers append, tail(1) and grep(1) work unmodified — but the data is stored in a backing directory as chunks compressed with zstd, together with a per-chunk write-time index that records when each chunk was written. Asking “what was written between 13:42 and 13:43?” is a binary search plus a few frame decompressions, independent of file size.

Files are append-only: writes anywhere but the end of the file fail with EPERM. Truncating a file to zero is allowed and starts it over (copytruncate-style rotation); rename and unlink work normally. ls -l shows the logical (uncompressed) size while du shows the real, compressed disk usage.

SUBCOMMANDS

  mount
timberfs mount BACKING MOUNTPOINT serves the logical view of backing directory BACKING on MOUNTPOINT, in the foreground. Unmount with fusermount3 -u MOUNTPOINT.

--chunk-size BYTES
    Uncompressed buffer size that triggers a chunk flush (default 262144).

--level N
    zstd compression level (default 3).

--flush-age SECONDS
    Maximum time appended data may sit unflushed (default 5). Bounds both the time granularity of the index and the data lost in a crash without fsync(2).

--allow-other
    Let other users access the mount. Unprivileged users need user_allow_other in /etc/fuse.conf; root (e.g. the systemd unit) does not.

  append
timberfs append FILE reads standard input and appends it to a log in the backing store directly, with no FUSE mount involved — the svlogd(8)/s6-log pattern:

    myapp 2>&1 | timberfs append logs-backing/app.log

Each log has exactly one writer (an exclusive per-file lock), but appenders for different files happily share a backing directory; a directory served by a timberfs mount is unavailable to appenders and vice versa. Appending to an existing log continues it. Data is flushed into chunks by the same size and age rules as the mount; end of input, SIGTERM or SIGINT flush and sync everything before exit. Accepts the same --chunk-size, --level and --flush-age options as mount, plus:

--retain DURATION
    Continuously drop data older than DURATION (e.g. 30d, 12h, 90m).

--retain-size SIZE
    Keep the on-disk (compressed) size of the log at or under SIZE (e.g. 200G, 512M; powers of 1024), dropping the oldest data first. Combines with --retain: whichever limit bites first wins.

Retention is checked every second, at startup and once more at shutdown. Dropping the head currently compacts the log by rewriting the remaining data, so enforcement is batched: age-expired data is dropped once it makes up about a tenth of the file, and a size overrun is trimmed down to 95% of the budget. Compaction briefly needs free disk space proportional to the retained data.

  import
timberfs import SOURCE... DEST converts existing plain log files into a timberfs log (the destination is the LAST argument). Several sources — a rotated set — may be given in any order: each file's first parsed timestamp determines the stitch order (rotation numbering and glob order lie), the plan is printed before anything is written, and stitching plain files requires an empty destination. Two sources starting at the same timestamp are refused as likely duplicates.

A source that is itself a timberfs log — a rotation segment shipped from another machine, say — is detected automatically (a .trunk/.rings path, or a logical name whose pair exists) and its chunks are merged verbatim into the destination: no decompression, no timestamp parsing, index included, cost proportional to the compressed size. A segment whose exact chunks are already in the destination is skipped, so re-running a shipping script is a no-op; merging refuses to break the destination's time ordering (import segments oldest-first). Plain and timberfs sources can be stitched together. Chunk time windows come from timestamps parsed from the log lines (historical data's write time is meaningless), so query works by the times the log actually talks about. Auto-detected timestamps must sit at the start of the line: RFC 3339 / ISO 8601 and common variants (space instead of T, dotted dates, and ./,/: before the milliseconds — logback's yyyy.MM.dd HH:mm:ss:SSS included) and leading epoch seconds/milliseconds; Apache/CLF ([10/Jul/2026:09:23:45 +0200]) is the one mid-line exception. Anything else via --timestamp-regex (one capture group, searched — anchor with ^ if wanted) plus --timestamp-format (chrono strftime). Zoneless timestamps are taken as local time unless --utc. Lines without a timestamp (stack traces, continuations) inherit the previous line's, so multiline entries land in the right window; a file that STARTS mid-entry backfills its head lines with the first timestamp found, and in a stitch they inherit the previous file's last stamp — exactly right for entries split by rotation. Mildly out-of-order lines widen chunk windows instead of getting lost. Accepts --chunk-size and --level.

Re-importing is idempotent: a non-empty destination is treated as a checkpoint. The already-imported bytes are verified against the same source ranges (every chunk, or first/middle/last with --quick), then only what the source has grown by is appended; an identical source is a no-op, and a source that shrank or changed is refused before anything is written (rotated files should be imported to a new target).

  export
timberfs export SOURCE DEST copies the chunks overlapping the given window (default: everything) into a NEW timberfs log — verbatim, no recompression, cost proportional to the compressed size of the window. The source needs no lock and may be live. A DEST ending in .timber writes the single-file transfer bundle instead: a plain uncompressed tar (the payload is already zstd) holding the .rings member first and the .trunk member second. Bundles are first-class READ-ONLY logs: query, index and export operate on a .timber file directly (tar stores its members contiguously and uncompressed, so the trunk member is simply a trunk at an offset), and import merges them — unpacking is only ever needed to append. Stock tools always suffice to recover one: tar xf x.timber && zstd -dc x.trunk (a pair tarred by hand follows the same convention). Export always creates; merging into an existing log is import's job.

  query
timberfs query FILE prints the bytes stamped inside the given time window to standard output, reading the backing files directly — it works with or without an active mount, but only sees flushed chunks (the still-buffered tail, at most flush-age old, is visible only through the mount). FILE is a backing file: the logical name, or its .trunk or .rings path.

--from TIME
    Start of the window (default: beginning of the file).

--to TIME
    End of the window (default: end of the file).

--has TOKEN
    Only chunks that (probably) contain TOKEN, tested against the .grain Bloom index (build it with timberfs reindex; without one, --has warns and scans the window). Matching is by WHOLE token — ASCII-alphanumeric runs of 3–64 characters, exact case; substrings do not match. An argument containing separators (req-8f3a) must match all its tokens in the same chunk; repeat --has for chunk-level AND. About 1% of non-matching chunks pass anyway (Bloom false positives) — the exact, entry-level filtering belongs in timberfs grep downstream. The killer use: finding a unique identifier with NO time bound.

Several FILE arguments give the fleet view over per-stream logs: chunks are selected per file (each file's own rings and grain) and interleaved across files by their time windows, with within-file order preserved; every output line carries a grep-style “path:” prefix (--no-filename to suppress). Attribution lives in the filename — store one log per host/app and merge at read time.

Selection is chunk-granular by design: every chunk whose write-time window overlaps the requested range is emitted in full, so the result carries at most flush-age worth of slop at each edge. The intended workflow is to let query do the coarse seek into a huge file, then trim exactly with grep(1) on the timestamps the log lines carry anyway.

  grep
timberfs grep PATTERN [FILE] is an entry-aware grep: a log ENTRY is a line carrying a timestamp plus all its continuation lines (stack traces, wrapped output), detected with the same timestamp auto-detection as import (and the same --timestamp-regex/--timestamp-format escape hatch). The pattern — a regex, or a fixed string with -F — is matched against the whole entry, and matching entries print whole; pipe several grep invocations for entry-level AND. Flags: -i (case-insensitive), -v (print non-matching entries), -c (count matching entries).

Input is a raw log on standard input, a plain log file, or a timberfs log/bundle — in the timberfs case --from/--to/--has pre-select chunks first (time index and .grain Bloom filters), the selection is padded one chunk each way so entries straddling a boundary arrive whole, and the pattern then matches exactly:

    timberfs grep req-8f3a backing/app.log --has req-8f3a
    cat any.log | timberfs grep 'tenantId=FOO' | timberfs grep -v DEBUG
    timberfs grep req-8f3a collector/*.log --has req-8f3a

Several files process in argument order with grep-style “path:” line prefixes (--no-filename to suppress) and per-file counts under -c.

  index
timberfs index FILE prints the chunk index of a backing file: per chunk the uncompressed offset and length, compressed length, compression ratio, and first/last write time, plus totals.

  reindex
timberfs reindex FILE builds (or rebuilds) the .grain token index for a log: one Bloom filter per chunk over every distinct token in it, ~10 bits per token, ~1% false positives — roughly 1–2% of the raw data size. Derived data under the sidecar contract: safe to delete, cheap to rebuild, and rotation/retention drop it (rebuild afterwards). A grain that lags a growing log is still correct — uncovered chunks are simply scanned.

  rotate
timberfs rotate --cutoff TIME SOURCE [DEST] moves every chunk written entirely before TIME out of SOURCE and appends it to DEST (created if missing, must be in the same backing directory). Compressed frames are relocated verbatim — nothing is recompressed — so the cost is proportional to the compressed size. A chunk straddling the cutoff stays in the source. Appending to an existing DEST is refused if it would break the destination's time ordering.

Rotation auto-detects a live mount: if a daemon serves the backing directory (advertised via the lock file), the request is routed through it and performed atomically; otherwise the backing files are rewritten directly under the same lock.

--delete
    Drop the rotated chunks instead of moving them (retention). Mutually exclusive with DEST.

--dry-run
    Show what would move without changing anything.

TIME FORMATS
TIME arguments accept RFC 3339 (2026-07-10T13:42:00+02:00), YYYY-MM-DD HH:MM[:SS] (local time, a T separator also works), HH:MM[:SS] (today, local time), and unix epoch seconds or milliseconds.

ON-DISK FORMAT
Each logical file name is backed by two files in the backing directory:

name.trunk
    The data: a plain concatenation of zstd frames, one per chunk, with no wrapper bytes. The full uncompressed content is therefore always recoverable with stock tools: zstd -dc name.trunk

name.rings
    The index: an 8-byte magic “RING0001” followed by fixed-size 48-byte records (little-endian u64 fields: uncompressed start/length, compressed start/length, first/last write time in unix milliseconds), appended in write order.

Chunks are written data-first, index-second; on open, index records pointing past the end of the data are dropped and orphaned data bytes are overwritten. fsync(2) through the mount flushes the buffer as a chunk and syncs both backing files.

The daemon holds an exclusive flock(2) on backingdir/.timberfs.lock recording its mountpoint; offline rotation takes the same lock.

EXTENDED ATTRIBUTES
Files in the mount expose read-only metadata via getfattr(1): user.timberfs.chunks, user.timberfs.compressed_size, user.timberfs.first_write, user.timberfs.last_write. The attribute user.timberfs.rotate is a write-only control interface used internally by timberfs rotate.

SYSTEMD
The Debian package ships a template unit timberfs@.service: create /etc/timberfs/instance.conf defining BACKING, MOUNTPOINT and optionally EXTRA_OPTS, then systemctl enable --now timberfs@instance. Stopping the unit unmounts first, so the daemon flushes all buffers and exits cleanly. An example configuration is installed under /usr/share/doc/timberfs/examples/.

EXAMPLES
Mount and use:

    timberfs mount ./logs-backing ./logs &
    myapp >> logs/app.log
    tail -f logs/app.log

Or skip FUSE entirely and pipe:

    myapp 2>&1 | timberfs append logs-backing/app.log

Extract a time window from a huge log, then trim with grep:

    timberfs query logs-backing/app.log --from 13:42 --to 13:43 | grep ERROR

Daily rotation and 30-day retention, no recompression:

    timberfs rotate logs-backing/app.log app-$(date -d yesterday +%F).log \
        --cutoff "$(date +%F) 00:00"
    timberfs rotate logs-backing/app.log --delete \
        --cutoff "$(date -d '30 days ago' +%F) 00:00"

Disaster recovery with stock tools only:

    zstd -dc logs-backing/app.log.trunk > app.log.recovered

EXIT STATUS
0 on success, non-zero on any error (2 for command-line usage errors).

SEE ALSO
fusermount3(1), zstd(1), getfattr(1), systemd.unit(5)

Project page and full design notes: https://github.com/torstei/timberfs