Performance

zetl is designed for large vaults — tens of thousands of notes — without sacrificing startup speed or interactive responsiveness.

(given fast-startup)
(given incremental-rebuild)

^perf-goals

Requirements

zetl must cold-start in under 200 ms on a vault of 10 000 Markdown files, complete incremental re-index in under 50 ms when fewer than 100 files have changed, and serve graph queries in under 10 ms at p99 on a fully-indexed vault of 50 000 nodes.

^perf-numbers

These figures were chosen to keep the tool imperceptibly fast during normal editing workflows, where the user expects results before they finish reading the command prompt.

^perf-rationale

Strategies

Incremental parsing

The Scanner only re-parses files whose mtime has advanced since the last run. On typical editing sessions (1–5 changed files), this cuts scan time by 99 % compared to a full rebuild.

^incremental-parsing-detail

Merkle-based change detection

BLAKE3 content hashes let zetl skip unchanged files even when mtimes are unreliable (e.g. after a git checkout). Each file’s Merkle root is stored in the Cache alongside its mtime.

; Merkle hashing enables reliable drift detection
(normally r-merkle-reliable
  (and content-addressed-hashing disposable-cache)
  reliable-change-detection)

^merkle-strategy

Memory layout

The link graph is stored as a flat adjacency list rather than a pointer-linked structure. This improves cache-line utilisation during multi-hop traversal and avoids GC pressure (irrelevant in Rust, but good discipline).

^memory-layout-detail

Targets summary

Metric	Target	Measured (10k vault)
Cold-start full scan	< 200 ms	140 ms
Incremental re-index (1 file)	< 50 ms	8 ms
Graph query p99	< 10 ms	4 ms

^perf-targets-table

See also: Cache, Scanner, Redis vs Memcached, Link Graph

Backlinks

Linked Pages

Link Graph

zetl builds a directed graph where nodes are pages and edges are Wikilinks. The graph powers all link-based queries: forward links, backlinks, multi-hop traversal, shortest path, and orphan detection.

Data model

Node — a Markdown page, identified by its title (filename without .md)
Edge — a wikilink from one page to another, with optional alias, heading, or block reference

The graph is built by the Scanner and cached by the Cache for incremental re-scans.

(given directed-graph)
(given multi-hop-traversal)

Queries

Command	Description
`links`	Forward links from a page, with configurable depth
`backlinks`	Pages linking to a target, with depth traversal
`path`	Shortest link path between any two pages
`export`	Full graph as JSON

These are exposed through Graph Queries. With the --with-conclusions flag, link results are cross-referenced with the Reasoning Engine to show what each linked page contributes to the vault’s logic.

Library

Built on petgraph, a Rust graph data structure library.

See also: Scanner, Graph Queries, Vault Diagnostics

Redis vs Memcached

Redis vs Memcached

We evaluated Redis and Memcached as caching backends for the zetl index layer. The decision was driven by benchmark data and operational simplicity requirements.

(given redis-evaluated)
(given memcached-evaluated)
(given single-node-deployment)

^cache-decision-context

Benchmark Results

We ran read/write throughput tests at various concurrency levels against both backends using a representative workload of 10 000 graph index lookups.

Backend	Reads/sec (p50)	Reads/sec (p99)	Writes/sec	Memory overhead
Redis 7.2	185 000	210 000	92 000	~12 MB
Memcached	170 000	198 000	88 000	~8 MB

^benchmark-results

The p99 numbers show Redis edges out Memcached on read latency under load, primarily because Redis pipelines responses more aggressively over a single connection.

^benchmark-interpretation

Analysis

Given the benchmark data, Redis meets our throughput target of 150 000 reads/sec with comfortable headroom.

; Redis read throughput exceeds our threshold — grounded to the benchmark table
(meta redis-fast-enough (source "^benchmark-results"))

Scanner

The scanner is zetl’s entry point for understanding a vault. It walks every Markdown file and standalone .spl file, extracting Wikilinks and Spindle Lisp blocks in a single pass.

Wikilink extraction

The scanner recognises all common wikilink forms:

[[Page]] — basic link
[[Page|alias]] — aliased link
[[Page#heading]] — heading anchor
[[Page^block-id]] — block reference
![[Page]] — embed

Extracted links feed into the Link Graph for query and traversal.

SPL extraction

Fenced code blocks tagged ```spl are extracted verbatim, along with their source file and line number. Standalone .spl files anywhere in the vault are also picked up. Both feed into the Reasoning Engine with full Provenance.

(given wikilink-extraction)
(given spl-extraction)

Implementation

Built on pulldown-cmark for Markdown parsing and ignore for .gitignore-aware file walking. The scanner is deliberately read-only — see Local-first Design.

See also: Cache, Link Graph, Reasoning Engine

Cache

Cache

zetl uses mtiame-based incremental caching for both the Link Graph and the Reasoning Engine’s theory. On subsequent runs, only files modified since the last scan are re-parsed.

How it works

The cache lives in .zetl/ at the vault root:

index.json — serialised link graph
theory.json — serialised reasoning theory and conclusions

Each entry records the file’s last-modified timestamp. When zetl starts, it compares mtimes and only re-scans changed files. Use --no-cache to force a full rebuild.

(given mtime-based-cache)
(given incremental-rebuild)

Design tension

There is an unresolved design tension in how aggressively to cache reasoning results:

; Cache the theory for fast startup
(normally r-cache-theory
  mtime-based-cache
  cache-reasoning-results)

; But recomputing ensures results are always fresh
(normally r-recompute-theory
  incremental-rebuild
  (not cache-reasoning-results))

Both arguments have merit. Try zetl reason conflicts on this vault to see how zetl surfaces this kind of tension. A future Plugin System could let users configure the trade-off.

Safety