- mark
- struck
- written in
- Zig
- license
- Apache-2.0
- version
- 0.1.0
- install
- zig build -Doptimize=ReleaseFast
Indexed regex search for a tree that changes while you are searching it.
The engines were not the problem. The consumer changed. Every tool in this field was built for a human typing a query every few minutes and reading the results with their eyes, who will notice if they look stale. What we have is roughly ten coding agents sharing one working tree, issuing many small searches per session, editing between those searches, and paying for every returned line in tokens. Three consequences fall straight out:
- Freshness stops being a nicety and becomes correctness. An indexed search that hands back a line which no longer exists has not been fast, it has lied, and an agent has no eyes to catch it with.
- A set is the wrong shape for an answer. A human scanning two hundred
hits finds the definition instantly; an agent reads them in order and burns
its context on call sites. Hence
--rank, which only ever reorders a set that was already verified. - Output is a budget. Every line costs money, which makes terseness a feature and the docs-versus-code partition a real axis rather than a convenience.
So the accelerators here are allowed to decline constantly and never allowed to
invent. A persisted trigram index and a resident session can save work; neither
may conjure a file set or hand back a stale line, and doubt of any kind returns
the query to the live walk. --no-index forces that walk, and it is the
differential oracle the indexed path is tested against. The tree tells the
truth.
Parity with ripgrep is a constraint rather than a resemblance, because that
muscle memory is already embedded in how agents search: the argv, the ignore
precedence, the exit codes and the bytes on stdout are ripgrep's, measured on
every run against a live rg oracle instead of asserted here. Where gist
diverges it is an improvement - identical-or-superset results, better in
behavior, performance or robustness, never a regression - or it is a bug. There
is no third category, and the eight cases are an enumerable bucket in
gist --schema rather than a paragraph that can drift from the binary.
Most of the machinery is inherited and cited: Thompson's linear matcher for
patterns that cannot be allowed to detonate, a vendored PCRE2 for the ones that
are not regular at all, Cox's trigram construction for deciding what never has
to be read. The one piece of mathematics that is ours is the crest sieve, a
sound necessary condition on forced class runs. It prunes precisely what the
whole trigram family concedes - the literal-free class repetitions like
[0-9a-f]{12} that a hash or MAC address hunt is made of - and it is sound by
construction rather than by testing, since every term in the bound rounds down
and any construct the calculus cannot certify contributes nothing.
Each generation of this tool found a different thing not to look at. Haertel's grep refused to touch every byte; Cox's index refused to open the file. This one adds the refusal they left on the table: not looking again at what has not changed.
- 33.7x
- warm geomean vs ripgrepDominance and fit certificate
- 19.4x
- warm geomean vs csearchDominance and fit certificate
- 10.4%
- index size against the text it indexedDominance and fit certificate
What this does not prove
- Memory is the lane gist loses. 4.56 GiB peak RSS while indexing, 2.7x zoekt and 1.6x csearch. It is the real scale ceiling in that table and it is not normalized away.
- The scan reaches 61.6 GB/s, 77% of the 79.8 GB/s single-core pure-read roof. That is below the pre-registered 80% threshold, so DRAM saturation is not certified and material headroom remains.
- Without an index gist is beaten on cheap literals. csearch answers literal-rare in 0.6x and regex-dotted in 0.5x of gist’s time, and zoekt takes literal-punct2 at 0.9x.
- Against csearch’s own query formula gist admits 10.5% fewer candidate bytes, but that is 6 classes won out of 20. The other 14 tie, because no planner can separate them.
Documents
- Technical reportGist is the tool we developed at the Billy Company to replace ripgrep as our default grep. It is indexed code search for a working tree that refuses to stay still: accurate while many agents write into the same corpus, faster without making an index authoritative, and shaped for a reader that pays for every returned line.
- Reading the numbersThe figures on this project are ratios against other search tools, and a ratio is easy to quote and easy to misread. This is what each word in them means, so you can decide what the certificate is worth before you open it.