English | 中文 | 日本語 | 한국어 | Tiếng Việt | Français | Русский

sage-wiki

sage-wiki is a graph memory and knowledge base that AI agents and humans build and query together. Drop in documents; an LLM compiler turns them into an interlinked wiki with a knowledge graph — agents query it through MCP, humans browse it as plain markdown. Enable the opt-in graph passes and it becomes an evidenced graph: typed entities, provenance-bearing relations, resolved aliases, and per-fact citations on answers. One Go binary scales it from a personal vault to a team hub to a company knowledge graph.

→ Get started: Install · Quickstart

Grown from Andrej Karpathy's idea of an LLM-compiled personal knowledge base, built with the Sage Framework. Some lessons learned along the way here.

• Graph memory with citations. Ask relational questions through wiki_graph_query — answers are grounded only in serialized graph edges; with the evidenced graph enabled, each citation carries its source document and confidence.
• Built for agents and humans. 19 MCP tools plus generated skill files teach agents when to search, capture, and compile; humans get Obsidian-native markdown, a TUI, and a web UI over the same data.
• Trust and provenance. Query outputs quarantine until verified; every evidenced relation records which document asserted it.
• Your sources in, a wiki out. The compile pipeline reads papers, notes, code, and email; summarizes; extracts concepts; and writes interconnected articles — the ingestion layer for everything above. Every new source enriches existing articles; the wiki compounds as it grows.
• Ask your wiki questions. Hybrid chunk-level search with LLM query expansion, re-ranking, and graph-aware context assembly returns cited answers.
• Scales to 100K+ documents. Tiered compilation indexes everything fast and spends LLM budget only where it matters.

https://github.com/user-attachments/assets/c35ee202-e9df-4ccd-b520-8f057163ff26

Dots on the outer boundary represent summaries of all documents in the knowledge base, while dots in the inner circle represent concepts extracted from the knowledge base, with links showing how those concepts connect to one another.

From personal vault to company knowledge graph

• Personal — overlay an existing Obsidian vault (init --vault), run on local models for zero cost, and opt into the graph passes (ontology.triples + ontology.resolve) when you want the evidenced graph.
• Team — share one wiki via git or a self-hosted server, review entity-resolution proposals and output trust together, and federate multiple wikis with the hub. See Team Setup.
• Company — move storage to PostgreSQL/pgvector, turn on metrics, front the server with auth, and scale ingestion with tiered compilation.

Knowledge graph & graph memory

Vector search retrieves passages that look like the query. A graph also records how things relate, so a question needing two or three hops is answered by traversal instead of hoping one chunk happens to contain the whole chain. sage-wiki builds that graph as a compile output — not a second database you have to keep in sync.

• Entities and typed relations. Each compile extracts entities (concepts, sources, artifacts) and links them with typed relations. The relation vocabulary is yours to define — see configurable relations.
• Evidenced edges. A relation can carry evidence (the span that supports it), confidence (0–1), and source_doc, so a conclusion traces to the sentence that justified the edge rather than to a whole document.
• Triples. An optional structured-output pass extracts subject → relation → object directly. Opt-in (ontology.triples): it adds one LLM call per document, and defaults never spend your key without asking.
• Entity resolution. "K8s" and "Kubernetes" become one node. Proposals are review-gated by default rather than silently merged.
• Concept curation. One opt-in pass (dedup_strategy: "llm") sees the whole proposed concept set at once — the only stage with global view — and judges keep / fold / drop per concept. Semantic restatements fold (aliases and sources merge), enumerated entities never fold (mw-3 is not mw-2, at any similarity), and drops stay logged proposals until llm_dedup.allow_drop opts in. The judgment prompt is workspace-overridable (prompts/curate-concepts.md).

The graph is a retrieval channel, not a side view. Every search fuses three channels — lexical (BM25), vector, and graph proximity: query terms seed entities, a bounded traversal ranks their neighborhood, and the three fuse at search.hybrid_weight_graph. An empty ontology costs nothing and leaves results byte-identical, so the graph earns its place incrementally.

Query it directly, or let an agent do it over MCP:

sage-wiki ontology query --entity kubernetes --depth 3 --direction both
sage-wiki provenance "service mesh"    # which sources produced this concept

Edges are bi-temporal: contradicting a fact invalidates the old edge instead of colliding, default answers are contradiction-free, and as_of queries answer "what did we believe in January?" Ambiguous contradictions still surface through output trust review. For corpus-wide questions ("main themes across everything?"), opt-in community detection (ontology.communities.enabled) generates cached community summaries and answers via wiki_graph_query mode: "global". Depth and mechanics: graph memory.

Guides

Install

# CLI only (no web UI)
go install github.com/xoai/sage-wiki/cmd/sage-wiki@latest

# With web UI (requires Node.js for building frontend assets)
git clone https://github.com/xoai/sage-wiki.git && cd sage-wiki
cd web && npm install && npm run build && cd ..
go build -tags webui -o sage-wiki ./cmd/sage-wiki/

Quickstart

Greenfield (new project)

sage-wiki init my-wiki && cd my-wiki
# Add sources to raw/
cp ~/papers/*.pdf raw/
# Edit config.yaml to add api key, and pick LLMs
sage-wiki compile                                  # first compile
sage-wiki compile                                  # second: zero LLM calls — unchanged docs are skipped
sage-wiki compile --explain raw/paper.pdf          # why a doc compiles or skips
sage-wiki compile --force                          # recompile everything regardless
sage-wiki search "attention mechanism"             # hybrid search
sage-wiki query "How does flash attention work?"   # cited Q&A
sage-wiki tui                                      # terminal dashboard
sage-wiki serve --ui                               # browser (webui build)
sage-wiki compile --watch                          # watch folder

Every config.yaml key, annotated line by line: Configuration.

Project layout (what init creates — selected entries, illustrative not exhaustive):

my-wiki/
├── config.yaml           # providers, models, compiler, search, ontology
├── raw/                  # drop sources here (articles, papers, code, images)
├── wiki/                 # compiled output — Obsidian-compatible markdown
│   ├── summaries/        # per-source LLM summaries
│   ├── concepts/         # concept articles (the knowledge graph)
│   ├── images/           # vision-captioned image descriptions
│   ├── outputs/          # filed query answers (trust.include_outputs: "true")
│   ├── under_review/     # filed answers awaiting trust review (default)
│   └── archive/          # pruned articles
├── .sage/wiki.db         # one SQLite file: FTS index, vectors, ontology, queue
└── .manifest.json        # source↔article mapping + compile state

Vault Overlay (existing Obsidian vault)

cd ~/Documents/MyVault
sage-wiki init --vault
# Edit config.yaml to set source/ignore folders, add api key, pick LLMs
sage-wiki compile --watch

Prefer containers? Prebuilt multi-arch Docker images and compose files are covered in the self-hosted server guide.

Supported Source Formats

Just drop files into your source folder — sage-wiki detects the format automatically. Images require a vision-capable LLM (Gemini, Claude, GPT-4o). Need a format not listed? sage-wiki supports external parsers — scripts in any language reading stdin, writing text to stdout.

Custom source types. Set type: on a source root in config.yaml and pair it with prompts/summarize-{type}.md to give any format — PDFs and Office documents included — its own summary prompt and source_type frontmatter. (image and code sources keep their built-in types — they select the vision and code pipelines, not a prompt variant.)

External source roots. A source path may point at an existing documents directory outside the project — Sage compiles it in place, no copy into raw/. Mark it read_only: true and ingestion refuses to write there. See configuration: external read-only source roots.

Graph memory

Out of the box the wiki builds a knowledge graph from keyword proximity — concepts linked where relation keywords co-occur with a [[wikilink]] in the same block. Enable the opt-in graph passes to turn that into an evidenced graph:

• Triple extraction (ontology.triples.enabled) — one extra LLM call per fully-compiled document extracts typed entities and relations, each carrying an evidence span, confidence, and source document.
• Entity resolution (ontology.resolve.enabled) — surface-form variants ("NASA" / "National Aeronautics and Space Administration") are linked to a canonical entity. High-confidence proposals apply automatically (threshold 0.85; set exactly 1.0 for review-only), and every link is exactly reversible with ontology resolve --unlink.
• Graph QA — the wiki_graph_query MCP tool answers multi-hop relational questions grounded only in a bounded, serialized set of edges; citations carry source_doc and confidence when the edge is evidenced (keyword-proximity edges carry neither). Regular Q&A context also names the connecting edge under each related article.

Depth, costs, review workflow, and undo semantics: Graph Memory.

Commands

The core surface; run sage-wiki --help for flags.

Topic-specific command families live with their guides: pack * in CONTRIBUTING, auth * (login, import, status, logout, migrate) in Subscription Auth, and verify / outputs * in Output Trust.

TUI

sage-wiki tui

A full-featured terminal dashboard with 4 tabs:

• [F1] Browse — Navigate articles by section (concepts, summaries, outputs). Arrow keys to select, Enter to read with glamour-rendered markdown, Esc to go back.
• [F2] Search — Fuzzy search with split-pane preview. Type to filter, results ranked by hybrid score, Enter to open in $EDITOR.
• [F3] Q&A — Conversational streaming Q&A. Ask questions, get LLM-synthesized answers with source citations. Ctrl+S saves answer to outputs/.
• [F4] Compile — Live compile dashboard. Watches source directories for changes and auto-recompiles. Browse compiled files with preview.

Tab switching: F1-F4 from any tab, 1-4 on Browse/Compile, Esc returns to Browse. Quit with Ctrl+C.

Web UI

sage-wiki serve --ui        # http://127.0.0.1:3333, requires -tags webui build

• Article browser with rendered markdown, syntax highlighting, and clickable [[wikilinks]]
• Hybrid search with ranked results and snippets
• Knowledge graph — interactive force-directed visualization of concepts and their connections
• Streaming Q&A — ask questions and get LLM-synthesized answers with source citations
• Table of contents with scroll-spy; dark/light mode with system preference detection; broken article links shown in gray

Built with Preact + Tailwind, embedded via go:embed (~1.2 MB, ~420 KB gzipped); omit -tags webui for a CLI/MCP-only binary. Auth tokens, allowed hosts, and deployment hardening: Self-Hosted Server.

MCP Integration

Add to .mcp.json (Claude Code; other agents in the Agent Memory Layer guide):

{
  "mcpServers": {
    "sage-wiki": {
      "command": "sage-wiki",
      "args": ["serve", "--transport", "stdio", "--project", "/path/to/wiki"]
    }
  }
}

Network clients: sage-wiki serve --transport sse --port 3333. The server exposes 19 tools — search, read, graph query, capture, wiki_query (question answering with trust-reviewed filing), compile-on-demand and more; setup per agent and capture workflows live in the Agent Memory Layer guide.

HTTP API (/v1, experimental) — any language can call the same tools over REST: sage-wiki serve --ui --port 3333 mounts 20 routes under /v1 (Bearer auth via SAGE_WIKI_TOKEN, structured errors, idempotent writes). Long-running compile/lint run as async jobs: POST /v1/jobs/compile or POST /v1/jobs/lint returns 202 + a job_id to poll. Contract: api/openapi.yaml (OpenAPI 3.1, drift-checked against the MCP tool set). Guide: docs/guides/http-api.md. Pre-1.0 — pin a version.

Agent skill files — sage-wiki skill refresh --target writes a behavioral section into the agent's instruction file (CLAUDE.md, .cursorrules, …) teaching it when to search, what to capture, and how to query, derived from your config. Targets: claude-code, cursor, windsurf, agents-md (Antigravity), codex, gemini, generic.

Agent skills

Install sage-wiki's reference skill so a coding assistant knows the full tool surface — all 19 MCP tools, the /v1 REST equivalents, opt-in flags, tiers, async compile semantics, and error codes — without reading this README:

# Claude Code
npx skills add https://github.com/xoai/sage-wiki --skill sage-wiki

# Or manually: copy skills/sage-wiki/SKILL.md to .claude/skills/

The sage-wiki-integrate pipeline skill wires sage-wiki into a new repo interactively (detect language → install client or configure MCP → smoke-test store-and-retrieve):

npx skills add https://github.com/xoai/sage-wiki --skill sage-wiki-integrate

Both skills are generated from the live MCP registry (go run ./tools/skillgen/) and drift-checked in CI — they cannot go stale when tools change. Pre-1.0 — pin a version.

Knowledge capture — agents store insights back via wiki_capture / wiki_learn, closing the read-capture-evolve loop. Workflows and tips: Agent Memory Layer.

Client SDKs

Typed clients over the /v1 REST API (pre-1.0 — pin a version):

Python — pip install sagewiki (≥3.9, httpx only):

from sagewiki import SageWiki

c = SageWiki()  # SAGE_WIKI_URL / SAGE_WIKI_TOKEN from env
for r in c.search("attention", limit=5).results:
    print(r.final_score, r.content[:80])
job = c.compile(topic="attention")
job.wait(timeout=600)  # explicit timeout required

TypeScript — npm install sagewiki (zero runtime dependencies, global fetch; Node ≥18, Deno, Bun, edge runtimes):

import { SageWikiClient } from "sagewiki";

const c = new SageWikiClient();
const results = await c.search("attention", { limit: 5 });
const job = await c.compile({ topic: "attention" });
await job.waitUntilDone({ timeoutMs: 600_000 });

Both clients cover the full /v1 surface: search, provenance, graph queries, the compiled wiki, captures/writes, and async compile/lint jobs with a code-driven error taxonomy. Docs: Python · TypeScript · HTTP API guide. Go programs can skip HTTP entirely — see Embedding in a Go program.

Examples

Copy-paste framework integrations, exercised in CI against a live server:

• examples/langgraph/ — memory-backed LangGraph nodes (Python client): retrieval with the uncompiled_sources → topic-compile pattern, plus capture.
• examples/vercel-ai-sdk/ — search, graphQuery, provenance as Vercel AI SDK tools (TypeScript client); edge-deployable.

Embedding in a Go program

To call the same tools from your own Go process — no subprocess, no stdio or port to manage — use pkg/sagewiki with mcp-go's in-process transport:

srv, err := sagewiki.NewServer("/path/to/wiki")  // project must already exist
if err != nil {
    return err
}
defer srv.Close()  // the caller owns the DB handle here

cli, err := client.NewInProcessClient(srv.MCPServer())
if err != nil {
    return err
}
defer cli.Close()

if err := cli.Start(ctx); err != nil {
    return err
}
if _, err := cli.Initialize(ctx, mcp.InitializeRequest{}); err != nil {
    return err
}

res, err := cli.CallTool(ctx, mcp.CallToolRequest{
    Params: mcp.CallToolParams{
        Name:      "wiki_search",
        Arguments: map[string]any{"query": "attention", "limit": 5},
    },
})

The project must already exist and the caller owns the database handle, so Close is required — unlike serve, nothing else closes it. Logs go to the host's stderr, and initialize reports the sage-wiki build version (dev from a plain go build); call sagewiki.SetVersion at startup to report your own version string instead.

The package is experimental while sage-wiki is pre-1.0: the Go signatures are meant to stay put, but tool names, argument schemas, and config.yaml layout can change in any release. Pin a version.

Operations

• Storage — SQLite by default (single file, zero config); PostgreSQL + pgvector for server deployments. Switching and pool sizing: Storage Backends.
• Observability — structured log snapshots and an opt-in /metrics endpoint: Metrics.
• Events & webhooks — the engine emits a typed event stream (captures, compile lifecycle, per-doc outcomes, graph edges, searches, LLM usage) to a rotating JSONL audit trail (events: block); serve mode adds HMAC-SHA256-signed webhooks and an SSE stream at /events/stream. Events never carry document content or raw query text. See Webhooks and Configuration § events.
• Structured outputs — LLM extraction passes use each provider's native mechanism (Anthropic tool-use, OpenAI response_format, Gemini responseSchema) with a validating fence-strip fallback.
• Credentials — subscription tokens live in the OS keychain where available; run sage-wiki auth migrate once to move file-stored credentials over. Subscription Auth.
• Configuration — every key, annotated, with multi-provider recipes and the serve-mode compile worker: Configuration.
• Entity resolution — 0.85 auto-apply, exactly reversible with --unlink; see Graph memory above.
• Custom relation/entity types — extend built-ins or add your own (ontology.relation_types), with multilingual synonyms and type restrictions: Configurable Relations.
• Output trust — query outputs quarantine until grounded, confirmed by consensus, or manually promoted: Output Trust.
• Search tuning — chunking, expansion, re-ranking, graph expansion, and opt-in ANN: Search Quality.

Embedding (Go API)

sage-wiki can be embedded as a Go module — the same engine the CLI runs on, workspace-scoped and offline-capable:

w, err := engine.Open(ctx, dir)          // one Workspace per directory
defer w.Close()
id, _ := w.Capture(ctx, engine.Source{Path: "doc.md"})
res, _ := w.Compile(ctx, engine.CompileRequest{Selector: "pending", Tier: 3})
hits, _ := w.Search(ctx, engine.SearchRequest{Query: "attention", Limit: 5})

pkg/engine is the only supported surface (facade over internal/): exclusive workspace lock (read-write Open fails fast with ErrLocked; WithReadOnly for lock-free reads), v0.2.x workspaces open read-only until adopted via WithUpgrade, and no internal/ type leaks into exported signatures (lint-tested). Companion packages: pkg/provider (LLM/embedding abstraction + providerfake for offline tests), pkg/events, pkg/mirror. A full offline example lives in examples/embed and runs in CI. The compile, search, capture, and query commands themselves route through pkg/engine. API stability note: pre-1.0, pin a version.

Cost

sage-wiki tracks token usage and estimates cost for every compile. Prompt caching (default on) reuses system prompts across calls within a compile pass — Anthropic and Gemini cache explicitly, OpenAI caches automatically — saving 50-90% on input tokens. Batch API (Anthropic, OpenAI, and Gemini) halves cost for large compiles:

sage-wiki compile --batch       # submit batch, checkpoint, exit
sage-wiki compile               # poll status, retrieve when done

compile --estimate previews cost; compiler.mode: auto batches automatically past a threshold. Prices come from a provider:model registry (embedded estimates → ~/.sage-wiki/prices.json → compiler.price_table); models with no entry report unknown, never a guessed number. Audit and review spend:

sage-wiki cost models     # which price produced a number, and its source
sage-wiki cost report     # recorded spend by model and pass/tier

Details: Configuration.

Resource limits

A single limits: block caps every ingestion, compile, query, and serve surface. Zero (unset) values resolve to the defaults below — a zero never means "disabled". Every violation fails fast with a typed error and emits a limit_exceeded event. Threat model and residual risks: Security.

limits:
  max_doc_bytes: 10485760                  # 10 MiB — max size of one ingested doc
  max_docs_per_capture_batch: 10           # max docs per capture batch
  max_compile_batch: 1000                  # max docs per compile run
  max_query_bytes: 32768                   # 32 KiB — max question length
  max_graph_traversal_nodes: 10000         # max nodes per graph traversal
  max_concurrent_provider_calls: 20        # concurrent LLM/embed calls in a compile
  max_concurrent_requests_per_conn: 8      # serve per-connection in-flight cap
  provider_timeout: 120s                   # per-call LLM/embed deadline
  compile_doc_timeout: 15m                 # per-doc compile budget

Serve mode is additionally hardened at the HTTP layer: request/header/idle timeouts, a 1 MiB request-header cap, a per-connection in-flight request guard (429 on breach), and a 1 MiB body cap on /mcp. An operator-owned rate-limit middleware slot (serve.Config.RateLimit, token-bucket example included) is the hook for deployment-level policy.

Scaling to large vaults

Tiered compilation routes each source by type and usage instead of LLM-compiling everything:

For large vaults: index everything at Tier 1 (a 100K-doc vault in ~5.5 hours), then compile on demand — auto-promotion, backpressure, and code parsers are covered in Large Vault Performance.

Bounded vector memory (opt-in)

By default every search loads the full vector matrix into memory per open workspace (vectors.backend: memory). For very large vaults — or many workspaces in one process — an on-disk, mmap-served index keeps resident memory bounded instead:

vectors:
  backend: mmap        # memory (default) | mmap
  quantization: none   # none (default, fp32 exact) | int8 (4x smaller)

Then build the index once (and after any compile/re-embed — a stale snapshot falls back to the in-memory cache with a warning, never to wrong results):

sage-wiki index rebuild-vectors

• fp32 (quantization: none) returns results identical to the in-memory backend (verified against the golden parity corpus).
• int8 is 4x smaller with a measured recall trade-off (recall@10 = 0.994 on the reference fixture; the gate is ≥ 0.95).
• The memory ceiling is unix-only (real mmap); other platforms serve the index from memory and warn once.
• Measured on a 50K×384 fixture: search heap 1.9 MB (mmap) vs 86.8 MB (memory) — ~2% resident; warm latency within 1.1x; cold search ~6x faster (no full cache load).

Multi-workspace serve

One process can serve many workspaces (personal/work/project vaults — and the seam a hosting setup composes around; this repo stays workspace-scoped, with no tenant/quota concepts):

sage-wiki serve --workspace-root /path/to/vaults --addr 127.0.0.1:8484

• GET /v1/workspaces lists the registry (subdirectories of the root containing a valid workspace).
• /w/{name}/... serves the full per-workspace surface (REST + MCP at /w/{name}/mcp). Workspace names are validated ([a-z0-9][a-z0-9-_]{0,63}) — invalid or unknown names get the same 404.
• Workspaces open lazily; --max-concurrent-compiles bounds compiles across all workspaces (one shared gate). Live workspaces are LRU-bounded by the engine Manager (engine.WithMaxOpen for API users).
• Auth: the root --token/--token-file guards all /w/* routes. Per-workspace auth is a non-goal — compose it in front (reverse proxy) if you need it.
• --workspace-root is HTTP-only and cannot be combined with --ui, --workspace, or --transport.
• Memory note: a served workspace opens up to 3 handles (engine, worker, MCP); the bounded-memory goal in multi-workspace serve requires vectors.backend: mmap.

Remote mirror (S3 backup)

Continuous, crash-safe replication of a workspace to any S3-compatible bucket (S3, R2, MinIO) and fast restore from it. The local directory stays the operating surface — the mirror is durability and mobility, never a live query path.

# 1. Configure the mirror: block (below), then:
sage-wiki mirror enable        # validates creds, writes manifest, bootstraps generation 1
sage-wiki mirror status        # local + remote state, pending changes, lag
sage-wiki mirror snapshot      # force a new generation
sage-wiki mirror verify        # full re-hash invariant check (--fast for HEAD-only)
sage-wiki hydrate s3://bucket/prefix /path/to/empty-dir

How it ships: the serve process ships continuously (ship_interval); every CLI command runs a best-effort ship pass after it finishes (success or error). The db ships Litestream-style (snapshot + WAL segments); markdown, prompts, sources, manifests, and vector indexes ship as content-addressed objects. Crash safety: the commit pointer is written last, so any kill leaves the previous committed state restorable — mirror verify proves it (full re-download re-hash; --fast is existence-only). Point-in-time restore: hydrate --at 2026-08-01T12:00:00Z (segment granularity for the db; overshoot printed); --generation N pins a generation; --partial restores in order (manifest → db → markdown → vectors) so lexical/graph works before vectors finish. PITR scope: the db restores at the selected generation/segment; markdown and source objects restore from the same generation's sealed object map (rotation points seal each generation's doc set into its meta.json) — so a rotated-generation restore is a consistent tree, not db@TIME with docs@newest. Docs restore at per-generation granularity: a mid-generation delete may still be present, a mid-generation create may be missing (bounded, and the restore report prints both skews). Mirrors written before object maps fall back to docs@newest with a printed note.

Credentials come from the environment (names configurable), never the workspace or config values:

mirror:
  enabled: false              # `mirror enable` sets this
  endpoint: ""                # e.g. https://.r2.cloudflarestorage.com or http://localhost:9000
  addressing: "auto"          # auto = virtual-host for amazonaws.com, path-style otherwise; "path"/"virtual" force
  bucket: ""
  prefix: ""                  # default: workspace directory name
  region: "auto"              # SigV4 region; "auto" works for R2/MinIO
  access_key_env: "AWS_ACCESS_KEY_ID"    # NAME of env var, never the value
  secret_key_env: "AWS_SECRET_ACCESS_KEY"
  session_token_env: "AWS_SESSION_TOKEN" # STS session token env var NAME (empty = absent)
  credentials_file: ""        # optional JSON {"access_key","secret_key","session_token"} outside the workspace
  ship_interval: "1s"         # WAL seal cadence while active
  snapshot_interval: "1h"     # scheduled generation cadence
  min_rotation_interval: "60s" # debounce for fold-forced rotations
  ship_lock_timeout: "5s"     # ship-mutex wait for CLI passes
  drain_timeout: "10s"        # serve shutdown budget for the final ship pass
  retain_generations: 2       # PITR depth in ROTATION COUNT, not time — raise for PITR-heavy use
  max_consecutive_defers: 10  # busy-writer deferrals before status surfaces rotation_deferred
  encryption:
    enabled: false            # AES-256-GCM client-side encryption
    key_file: ""              # 32-byte key file — MUST live outside the workspace

Notes:

• config.yaml itself is not mirrored (it can hold secrets like api.api_key) — hydrate restores data only; run sage-wiki init or restore your own config after a migration.
• Optional encryption is AES-256-GCM with a keyfile; mirror verify works without the key (integrity hashes cover shipped ciphertext).
• The standalone sage-wiki tui has no in-process shipper — its changes ship at TUI exit (kill -9 → at the next command). serve and serve --ui ship continuously.
• RPO: with defaults, induced loss loses ≤ ship_interval of writes (measured: 30.8ms of writes at a 50ms interval — see the test TestRPO_ServeShipper).

Ecosystem

Contribution Packs

Packs bundle ontology types, prompts, and skill triggers for a domain. Eight bundled packs work offline:

sage-wiki init --pack academic-research applies one at init; pack install adds more. Creating and publishing packs: CONTRIBUTING.

External Parsers

Handle any file format with a script in any language (stdin → text on stdout), declared in parsers/parser.yaml behind a double opt-in — they run as unsandboxed subprocesses with timeout enforcement and environment stripping. Authoring and hardening details: CONTRIBUTING; the trust-boundary discussion: Team Setup.

Teams

Three sharing patterns — git-synced, shared server, hub federation — plus team trust review and cost management: Team Setup.

Benchmarks

Two suites answer different questions. Full detail: eval/benchmarks/REPORT.md and eval/REPORT.md.

Memory benchmarks — can it answer questions about a long conversation? Published datasets, LLM-judged, using the prompts and procedure from mem0ai/memory-benchmarks with sage-wiki as the backend (gpt-5 answerer/judge, scoped samples):

Not a like-for-like ranking: mem0 runs their managed platform on full question sets, these are scoped samples (±4–5pp), and the compile pipelines differ. The caveats are spelled out in the report.

Quality + performance eval — is the wiki well-formed and fast? Runs on any compiled wiki, no API keys, seconds. Median across 10 real wikis: overall 87.4%, fact extraction 100%, search recall@10 100%, cross-reference integrity 100%. In-process retrieval: FTS5 top-10 0.035 ms, hybrid RRF 4.9 ms, graph BFS 0.001 ms.

python3 eval/eval.py .                      # quality + perf on your wiki
python3 -m pytest eval/eval_test.py -q      # harness self-tests

Architecture

• Storage: SQLite with FTS5 (BM25 search) + BLOB vectors (cosine similarity) + compile_items table for per-source tier/state tracking
• Ontology: Typed entity-relation graph with BFS traversal and cycle detection
• Search: Unified pipeline — document- and chunk-level FTS5 and vectors fused by weighted RRF with the ontology graph as a third channel, corpus-adaptive stopwording, title-proxy column weights, and a recency tie-breaker on documents with a known origin date. LLM query expansion and coverage-gated re-ranking are opt-in per call on the search surfaces and on by default for Q&A, which also gets 4-signal graph context expansion. Search responses signal uncompiled sources for compile-on-demand.
• Compiler: Tiered pipeline (Tier 0: index, Tier 1: embed, Tier 2: code parse, Tier 3: full LLM compile) with adaptive backpressure, concurrent Pass 2 extraction, prompt caching, batch API (Anthropic + OpenAI + Gemini), cost tracking, compile-on-demand via MCP, quality scoring, and cascade awareness. Embedding includes retry with exponential backoff, optional rate limiting, and mean-pooling for long inputs. 10 built-in code parsers (Go via go/ast, 8 languages via regex, structured data key extraction).
• MCP: 19 tools (7 read, 9 write, 3 compound) via stdio or SSE, including wiki_graph_query for provenance-cited multi-hop graph QA, wiki_compile_topic for on-demand compilation and wiki_capture for knowledge extraction
• TUI: bubbletea + glamour 4-tab terminal dashboard (browse, search, Q&A, compile) with tier distribution display
• Web UI: Preact + Tailwind CSS embedded via go:embed with build tag (-tags webui)
• Scribe: Extensible interface for ingesting knowledge from conversations. Session scribe processes Claude Code JSONL transcripts.
• Packs: Contribution pack system with 8 bundled packs, Git-based registry, install/apply/remove/update lifecycle, transactional apply with snapshot rollback, fill-only merge, and config allowlist security.
• External Parsers: Runtime-pluggable file format parsers via stdin/stdout subprocess protocol. Sandboxed execution with timeout, env stripping, and network isolation (Linux).

Zero CGO. Pure Go. Cross-platform.

License

MIT