Spis
Spis
Spis/Docs

Architecture

Spis is a single Rust command-line process over a file-backed corpus. There is no database or resident service in the maintained path. src/main.rs collects arguments, src/commands/mod.rs dispatches subcommands, command modules own workflows

Overview#

Spis is a single Rust command-line process over a file-backed corpus. There is no database or resident service in the maintained path. src/main.rs collects arguments, src/commands/mod.rs dispatches subcommands, command modules own workflows, and src/lib.rs supplies shared HTTP, robots, sitemap, HTML extraction, hashing, time, and JSON helpers.

Components#

text
                    +-----------------------+
                    |      Rust spis CLI    |
                    +-----------+-----------+
                                |
        +-----------------------+-----------------------+
        |                       |                       |
 catalog/reference lane    docs-crawl lane        acquisition adapters
        |                       |                       |
 *-examples/*.json        ~/.spis/docs-corpus     gh / Stado / Weles /
 retained media           pages.jsonl.gz          installed product PTYs
 generated indexes        state.json

Catalog/reference lane

catalog_type, reference_record, and reference_contract manage source structures and shared vocabulary. verify_reference_evidence measures record bytes. generate_example_catalogs is the cross-file consistency gate and renderer. Structure, README, guideline, image, accessibility, and drift modules enrich or monitor those files.

Documentation-crawl lane

crawl_docs reads the checked-in inventory definitions, resolves URLs, and places work into a process-wide queue. Requests to one host are not spaced. Worker threads fetch and extract pages; one writer thread per site appends gzip JSONL; a periodic flusher writes resumable state through a temporary file and rename. docs_corpus is a read-only streaming reader over the same files.

HTTP layer

The documentation crawl's GET (http_get in src/commands/crawl_docs/override_group/validate_public_endpoint.rs) sends each request once, to the addresses the URL policy pinned, and ignores proxy variables in the environment. It follows every redirect itself, holding each target to the same URL policy, and refuses a redirect loop, a redirect without a Location header and a URL that changed without a redirect. The body is read whole; a failed request or read is reported with its cause. Robots rules are read per origin and kept as a snapshot.

Other modules fetch with their own clients: spis discover with its discovery user agent (src/commands/discover/ua.rs), which reads at most the first 64 MiB of a body; spis collect-example-images with the Spis user agent, reading each page and image whole; README drift through gh api against the GitHub contents API. Discovery ranking asks Brama which discovered pages to keep (src/commands/discover/rank.rs).

Real-product capture

capture_cli_references reads the capture plan named with --plan, searches PATH for each planned binary, forks /bin/bash --norc --noprofile -i in a pseudo-terminal of the plan's terminal.columns x terminal.rows, normalizes the environment, and executes a seven-step probe plan. Output is timed into an asciinema v2 cast. Five named points are rendered as PNG evidence at the plan's terminal.font_px. The capture is measured and written into the plan's catalog.

Fleet/browser boundary

Spis produces Weles capture/audit plans but does not SSH to hosts. It calls Stado to submit/poll work; Weles owns browser execution. Returned artifacts are verified before record metadata changes. This preserves provenance and keeps placement outside Spis.

Data ownership and generated views#

The source of truth is distributed deliberately:

  • sources.json owns why an example is selected and its overview/structure facts;
  • each reference.json owns product evidence and gaps;
  • retained files own the bytes that hashes and durations describe;
  • references.json, catalog README.md, and root catalog indexes are generated views;
  • $HOME/.spis/docs-corpus is local resumable crawl state, not checked-in interface evidence.

The generator refuses disagreement rather than selecting one conflicting copy silently.

Process and failure model#

The CLI is synchronous at the command boundary. Internal commands may use worker threads, subprocesses, or long-running Stado jobs. Ordinary errors return through anyhow and main prints error: with exit 1. Usage errors are a type, commands::Usage, and exit 2: unknown top-level commands, a subcommand's usage line, unknown arguments or actions, and missing or malformed flag values. Accessibility audit reserves exits 2/3 for planning versus execution/retrieval. Strict drift exits 1 directly when drift is observed.

Runtime boundary#

scripts/build-release.sh builds the Rust binary and .wisent-release.json reads the exact version from Cargo.toml; no alternate command dispatcher or package version source is retained.

spis serve is the loopback HTTP/JSON backend Spis Desktop starts; each of its endpoints runs the spis command it names as a child process, so the desktop client and the CLI share one data path (see `spis serve`).