NoKV

Durable, versioned workspaces for disposable agents.

NoKV keeps declared inputs, code, outputs, logs, and lineage alive when an
agent sandbox disappears. It publishes immutable S3-compatible artifact
bytes through a path-native, transactional metadata control plane; it does
not pretend to restore process memory, a model session, or a VM.





18-operation native CLI ·
direct Python and Rust SDKs ·
commit, snapshot, and restore ·
conditional publication and exact replay





[![Rust CI](https://github.com/NoKV-Lab/NoKV/actions/workflows/rust.yml/badge.svg?branch=main)](https://github.com/NoKV-Lab/NoKV/actions/workflows/rust.yml)
[![Python SDK](https://github.com/NoKV-Lab/NoKV/actions/workflows/python-sdk.yml/badge.svg?branch=main)](https://github.com/NoKV-Lab/NoKV/actions/workflows/python-sdk.yml)
[![Latest release](https://img.shields.io/github/v/release/NoKV-Lab/NoKV?sort=date&display_name=tag&label=release)](https://github.com/NoKV-Lab/NoKV/releases/latest)
[![Apache-2.0 license](https://img.shields.io/github/license/NoKV-Lab/NoKV)](./LICENSE)





[What it adds](#what-nokv-adds) ·
[Use cases](#use-cases) ·
[Feature map](#feature-map) ·
[Integrate](#choose-an-interface) ·
[Evidence](#evidence-and-qualification) ·
[Quick start](#quick-start) ·
[Docs](./docs/index.md)

[!IMPORTANT] This README describes the current main contract. An installed release may pin different component revisions; run nokv version --json to identify the exact NoKV commit, Workbench schema, and Holt build. Implemented and deterministic-test-passing do not mean live or production-qualified. The current qualification boundary is explicit below.

What NoKV Adds

Most agent systems already have an object store and a database. Their hard problem is the failure window between them: bytes may exist without a visible workspace entry, a retry may publish twice, a stale worker may overwrite a newer result, or cleanup may delete an object that history still references.

NoKV closes that window as one artifact lifecycle:

produce bytes
    |
    v
upload immutable blocks + verify provider admission
    |
    v
one final bounded metadata command makes the revision visible
  - compare the current path generation and owner epoch
  - publish the immutable revision and advance the path head
  - update indexes, references, change history, and operation receipt
    |
    v
visible workspace result, or a typed conflict / recoverable exact replay

The current hard-to-substitute increment is this combined closure, not one data structure or a claim that alternatives cannot exist. A replacement must preserve all of the following under response loss and restart, then pass the same conformance and fault evidence:

• Atomic visibility: artifact bytes are staged first and become reachable only when their revision, path head, indexes, references, and receipt commit.
• No silent overwrite: create, replace, edit, append, rename, and remove are explicit and generation-fenced. A stale writer gets a conflict.
• Exact retry: request identity and operation records distinguish definitely-not-applied, applied-with-lost-response, and unknown outcomes.
• Durable decision points: commits retain an immutable revision closure; restore builds a hidden same-root incarnation and publishes it atomically.
• History-aware collection: object listing is never namespace truth. GC is fenced by path references, commits, snapshots, restore operations, and quarantine state.
• Fenced ownership: persisted root placement and owner epochs prevent an obsolete shard owner from continuing to publish metadata.

Find a capability by job

Use Cases

State provider for disposable agent sandboxes

An agent sandbox should be disposable; its durable state should not be. Any agent, loop harness, or sandbox runner can keep transient execution local while using NoKV as the durable state provider for everything that must survive: task state, inputs, code, tool outputs, logs, checkpoints, and lineage.

run inside a sandbox
  -> publish immutable state and artifacts with generation fences
  -> seal a deterministic Workbench commit

sandbox exits or disappears
  -> reopen the workspace from its durable identity
  -> continue from the commit, or restore it into a fresh Workbench
  -> retry the same request exactly; reject an obsolete writer

The value is not another state-blob write. NoKV is not raw storage alone; it is an implemented, composable, sandbox-native functional storage layer. A harness can use its lifecycle primitives through the native CLI or SDK:

• Atomic checkpoint visibility: artifact bytes are published first; a Workbench commit then seals one canonical manifest and its exact immutable revision closure as the durable decision point.
• Idempotent recovery: for the same request identity and inputs, durable operation receipts return the prior outcome after a lost response instead of applying the same transition twice.
• Stale-writer rejection: a caller submits the generation it observed; generation compare-and-swap rejects its update if newer state won first. Owner epochs separately fence an obsolete shard owner.
• Restore without payload copying: a retained commit can populate a hidden same-root incarnation whose destination becomes visible atomically. Leased snapshots provide shorter-lived point-in-time recovery and inspection.
• One recoverable workspace: verified materialize/collect, indexed search, immutable lineage, and retention-aware GC share one workspace contract. The Rust SDK additionally exposes change polling; quarantine and reconciliation remain internal lifecycle mechanisms rather than CLI commands.

The harness still decides which state is authoritative and when execution may resume. Restoring state does not grant a replacement worker permission to act; the harness must reacquire its own leases and gates. NoKV recovers declared durable state, not RAM, sockets, credentials, container or VM state, or an in-flight model session. These lifecycle primitives are implemented and layer-tested at the evidence revision below; a complete harness adapter and authority handoff remain caller-owned and require separate live qualification.

AI-for-science: durable evidence for adaptive HPC loops

An adaptive AI-for-science campaign can use a fast surrogate to screen a broad candidate space, spend expensive high-fidelity compute on a selected subset, and feed the verified results into the next round. atomate2 and jobflow continue to own the scientific Jobs and Flows, dependencies, result schemas, JobStore, and execution environment. NoKV adds the durable state and evidence boundary between rounds and around the expensive jobs:

1. Publish exact inputs, code, configuration, model artifacts, and candidate sets as immutable, versioned artifacts.
2. Before high-fidelity work is launched, seal the selected tasks and decision inputs as one deterministic Workbench commit.
3. Record the workflow's calculation identity and attach returned outcomes, including failures, as new immutable revisions instead of overwriting prior evidence.
4. Retrain, re-analyse, or restore from a retained commit identity; populate a fresh Workbench from its closure without silently mixing incompatible inputs or results.

This target shape makes each expensive selection a reproducible checkpoint: what was chosen, from which inputs and model state, under which code and configuration, and what came back. NoKV does not build the workflow graph, submit or cancel external jobs, validate scientific correctness, or replace the workflow engine, scheduler, or JobStore. This is a target integration shape, not a currently bundled atomate2/jobflow adapter, partner adoption, or live-qualified HPC deployment. A future adapter would bind workflow job ids, output references, and scheduler receipts to Workbench revisions; scientific workflow semantics remain above NoKV.

Feature Map

Native Workbench CLI: the stable agent surface

nokv workbench '' is the primary integration surface. The names and normalized input schemas below are fixed by the Workbench contract.

All 18 operations have complete parser-to-metadata implementation chains, with object-store effects for body-bearing operations, and deterministic adjacent-layer tests at the evidence revision below. Their native-CLI real-service acceptance gate is still not qualified.

Other CLI capabilities

For an append that must survive a lost reply or a harness restart, save one operation id before sending the request, then reuse it with the same inputs:

nokv [route/agent/object options] workspace-path append run-42 logs events.jsonl \
  --operation-id aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa \
  --expected-workspace-incarnation-id bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb \
  --text 'step-7-done' --content-type text/plain

Replace the example ids with the caller's stable logical operation id and the intended workspace's incarnation. On the first call, omitting the incarnation observes and fences the existing workspace. On recovery, omission resolves the original durable operation first, even if the workspace was deleted or recreated. Append never creates a missing workspace.

Use exactly one of --text, --base64, or --file. Text defaults to text/plain; charset=utf-8 on creation; binary and file input default to application/octet-stream. Existing content type is inherited unless --content-type overrides it. Explicitly use the same type when replaying across text, file, and Python inputs. The delta limit is 16 MiB; --max-artifact-bytes may lower it. The resulting body also defaults to a 16 MiB limit; --max-logical-size raises that bound and is part of the stable intent. --block-size defaults to 4 MiB and matches Python's block_size argument; retain the same value across retries and surface changes.

The receipt identifies both the logical operation_id and its successful publication_operation_id. Its revision, generation, workspace revision, size, digest, and incarnation describe the original publication, even if another writer has advanced or removed the path. replayed and nullable commit_version describe this SDK call and are not stable receipt fields.

Query a result without resending the delta or contacting the object store:

nokv [route/agent options] operation status aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa

The response supplies a machine-readable recovery action:

Status also retains the current physical attempt's cause_code, failure_message, and structured attempt_failure, including when the logical operation is ready_to_retry.

Inspect and recover without object credentials:

nokv [route/agent options] operation inspect aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa --limit 32
nokv [route/agent options] operation recover aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa \
  --expected-state-digest 

Inspection returns operation_token and publication_token, each with operation_id and state_digest, plus the object namespace, registered_count, cleanup_cursor, remaining_count, and entries. Entries describe retained staged objects, not a historical inventory of retired keys. Each entry includes its sequence, identity, expected length, digest, and optional multipart token. The page limit defaults to 32 and may range from 1 to 192. Continue with the opaque base64 next_cursor using --cursor; a stale cursor returns Conflict/OperationState, requiring a fresh inspection instead of mixing pages.

Save the logical token before requesting recovery. Reuse its state digest after a lost reply or process restart. The owner performs sealing; this command never resends the append delta. requested=true means a cleanup request was durably accepted, including a replay. Its historical recovery_receipt contains the logical id, publication id, cleanup retry count, and expected state digest. The top-level status is a fresh observation and may be newer than that receipt; acceptance is not cleanup completion. Omitting the digest targets the current state once and is a no-op for pending, cleaned, or committed operations. A later quarantine needs an intentional new request with a newly inspected token. AppendCleanupUnresolved preserves the saved digest, any known recovery receipt, and next_action=retry_same_cleanup; generic retry handlers must not choose a new token. After ready_to_retry, resubmit the original append inputs.

An interrupted publication must be fenced and cleaned before a successor attempt can start. Status queries only observe this lifecycle. Persist the delta or a way to reproduce the exact bytes until the operation commits. An unknown result or NotFound observation is not proof that a concurrent request cannot commit; errors preserve the logical id and use next_action=query_same. cause_code=RequestReplayMismatch means the identity belongs to different inputs or a different lifecycle: recover the original intent before retrying. Never generate a replacement id merely to bypass an uncertain outcome.

The older workbench_append tool retains per-invocation identities and cannot recover one action across separate processes. Use workspace-path append for new retrying harness integrations. The fixed 18-tool contract remains unchanged. The append product specification defines the supported recovery, size, retention, and release boundaries.

Programmatic and non-CLI capabilities

The interfaces share protocol, metadata, object, and lifecycle semantics. They do not expose identical method shapes.

Python append_bytes, operation_status, operation_inspect, and operation_recover share the native stable append contract. Python inspection cursors are opaque bytes; recovery state digests are lowercase hex strings. The SDK does not expose the legacy 18-tool append facade, exact-string edit, or body grep methods; use the CLI for those exact behaviors. Custom SDK compositions are caller-owned and are not equivalent qualification evidence.

Choose an Interface

Downstream skill or shell harness
  -> native nokv CLI                 workspace operations and stable append recovery

Embedded Python application
  -> nokv Python Client/adapters     direct path/artifact/lifecycle API

Native control plane or provider
  -> nokv-client Rust crate          lower-level typed integration

Use the native CLI for agent skills because it provides one inspectable schema, stable JSON envelopes, and fail-closed admission. Use Python when the caller needs in-process range reads, fsspec/checkpoint integration, or direct typed methods. Use Rust for provider, routing, change-feed, or recovery-aware work.

Every agent-facing CLI command requires self-refreshing etcd routing plus the durable RootId to AgentId binding. Static route pins remain an SDK/testing option and are rejected by the agent-facing CLI.

Core Semantics

Namespace and publication

• RootId is the only storage and routing identity. A presentation path such as /agents/research/wb shapes returned paths and manifests but grants no storage authority.
• PathCurrent(root, incarnation, normalized_path) is canonical namespace truth. Directories are implicit prefixes; an object-store listing is never authoritative.
• Every published body has a never-reused artifact revision id, immutable revision-owned blocks, a whole-body digest, and a caller-visible generation.
• Append reads the current head and publishes an immutable generation-fenced revision. Stable append records a logical operation before publication; a successor attempt is allowed only after its predecessor cannot publish and has completed cleanup. One logical append has at most one committed result.

Commit, snapshot, and restore

• A commit is the durable decision point: it binds a canonical run manifest and retains the exact artifact revision closure.
• A snapshot is a leased liveness hold, not archival retention. Use snapshot plus restore while the lease is active; use commit plus restore when a decision point must remain recoverable after the lease expires.
• Restore preserves its source, reuses immutable payload blocks, stages a fresh same-root incarnation invisibly, and atomically publishes the new destination. It is not a cross-root or cross-shard copy transaction.

Sharding, identity, and authority

• Root placement is persisted control-plane state, never derived from a path prefix or the current shard count.
• One epoch-fenced active owner executes a shard-local metadata command.
• Transactions, snapshots, commit closure, restore, and GC barriers are shard-local. NoKV does not provide cross-shard transactions.
• Workbench jailing and RootId to AgentId binding are safety boundaries, not tenant authentication, authorization, or RBAC.

Why Holt Fits the Current Local Profile

Holt is NoKV's purpose-built embedded Rust metadata engine. Its persistent adaptive radix trees provide ordered point and prefix access for path-shaped metadata while its WAL and atomic batches provide the local durability boundary. NoKV owns the workspace semantics in nokv-meta over the storage-neutral TxnStore contract; Holt is the currently wired and tested LocalSync implementation, while S3-compatible storage still owns artifact bytes.

This evidence snapshot covers NoKV's exact holt = "=0.8.6" dependency. Later upstream Holt main features are outside this section's claims; identify an installed build with nokv version --json.

Holt maps NoKV metadata families to named persistent adaptive radix trees:

• adaptive Node4/16/48/256 fanout and path compression fit NoKV's prefix-rich encodings for workspace paths, commit members, and indexes;
• the same persistent ART index supports point lookup and native ordered prefix/start-after/delimiter pages, which maps directly to hierarchical listing without rebuilding order in the application;
• one bounded DB::atomic batch checks and mutates multiple metadata families through a single WAL record, matching one NoKV metadata command;
• one scoped DB::view captures a consistent multi-tree read view for a NoKV read batch; it is not the same object as a durable NoKV snapshot;
• NoKV's file-backed Holt profile forces synchronous WAL acknowledgement; replay, shadow checkpoint slots, and an exclusive manifest writer define the tested same-namespace restart boundary;
• compacted blobs support page-granular routed reads, while corrupt read accelerators fall back to the authoritative blob.

This is a workload fit, not proof that ART is universally faster or uniquely able to implement NoKV. The adapter contract deliberately permits another TxnStore after equivalent conformance, receipt, snapshot, scan, restart, and fault qualification. Current tests cover deterministic torn-frame injection and a real child SIGKILL after a durable non-empty partial manifest record; they do not constitute a physical power-cycle, remote failover, cross-host fencing, or metadata-HA test. No current third-party A/B number is claimed.

Evidence and Qualification

The labels below mean exactly:

• Implemented: the typed call chain and durable effect exist in current source.
• Layer-tested: deterministic tests exercise real adjacent implementations; fake/mock tests remain identified as such.
• Live-qualified: the supported entrypoint ran against the required real dependencies at the pinned revision and produced accepted evidence.

Evidence snapshot:

• NoKV commit 0f1995ebee96;
• pinned Holt 0.8.6 dependency.

The 10 ignored NoKV workspace tests are six real-etcd tests, two direct object provider tests, and two Python S3-admission tests. They are gaps, not passes. See Workspace acceptance for the normative gate and Benchmarks and evidence for evidence rules.

Implemented, but not currently live-qualified

• opt-in shared recovery publication and receipt-directed recovery-log install.

Not qualified by current evidence

• remote checkpoint compaction, copied-directory failover, shared metadata durability, multi-machine failover, or metadata HA;
• accepted full-surface native-CLI and installed-Python Gate 0 live acceptance;
• complete provider timeout and ambiguous-delete fault coverage;
• cross-host writer fencing;
• physical power-cycle behavior, production tail latency, write amplification, RSS advantage, or universal ART performance.

Not offered by NoKV

• tenant identity, authentication, or RBAC;
• cross-shard transactions;
• transparent POSIX/FUSE/CSI/NAS behavior.

Quick Start

Inspect the contract locally

Build the CLI, identify the exact bits, and inspect the frozen schema:

cargo build --release -p nokv --bin nokv
./target/release/nokv version --json
./target/release/nokv schema
python3 scripts/workbench/workbench_contract_test.py

These commands prove build identity and the offline contract. They do not exercise a deployed metadata owner or object provider.

A source Formula is available from NoKV's public Homebrew tap:

brew install NoKV-Lab/tap/nokv
nokv version --json
nokv schema

The Formula may trail the latest GitHub release. Read back nokv version --json and verify release-specific target evidence before treating a version or platform as supported.

Call a provisioned deployment

A live deployment requires a persisted root placement, a leased shard owner, etcd routing, and admitted S3-compatible object coordinates. After following the live deployment preflight, the current source-level CLI shape is below. The command form is documented; full native-CLI real-service Gate 0 remains not qualified.

nokv \
  --root-id "$NOKV_ROOT_ID" \
  --agent-id "$NOKV_AGENT_ID" \
  --workbench-root /agents/research/wb \
  --etcd-endpoint "$NOKV_ETCD_ENDPOINT" \
  --object-bucket "$NOKV_BUCKET" \
  --object-endpoint "$NOKV_OBJECT_ENDPOINT" \
  workbench workbench_create '{"id":"run-001"}'

nokv \
  --root-id "$NOKV_ROOT_ID" \
  --agent-id "$NOKV_AGENT_ID" \
  --workbench-root /agents/research/wb \
  --etcd-endpoint "$NOKV_ETCD_ENDPOINT" \
  --object-bucket "$NOKV_BUCKET" \
  --object-endpoint "$NOKV_OBJECT_ENDPOINT" \
  workbench workbench_put_file \
  '{"id":"run-001","section":"scripts","path":"main.py","text":"print(42)","replace":false}'

nokv \
  --root-id "$NOKV_ROOT_ID" \
  --agent-id "$NOKV_AGENT_ID" \
  --workbench-root /agents/research/wb \
  --etcd-endpoint "$NOKV_ETCD_ENDPOINT" \
  --object-bucket "$NOKV_BUCKET" \
  --object-endpoint "$NOKV_OBJECT_ENDPOINT" \
  workbench workbench_read \
  '{"id":"run-001","section":"scripts","path":"main.py","format":"structured"}'

Credentials use the provider's normal chain or --object-access-key-id, --object-secret-access-key, and optional --object-session-token; never embed them in Workbench content or manifests.

Architecture and Ownership

agent skill / harness       embedded Python         native control plane
        |                        |                         |
        v                        v                         v
  18-tool CLI facade      direct Python API       lower-level Rust SDK
        +------------------------+-------------------------+
                                 |
                         typed NoKV protocol
                                 |
                   epoch-fenced logical-shard owner
                                 |
                    nokv-meta state machine
                 path, revisions, receipts, holds,
                    queries, restore, and GC
                       /                   \
                      v                     v
        TxnStore metadata adapter      artifact object store
          current local: Holt          S3-compatible bytes

NoKV is not a semantic-memory database or an agent orchestrator. Retrieval ranking, planning, scientific validation, trace/scorer semantics, and runtime policy remain above it. The object provider owns physical durability, replication, availability, and access policy for bytes. NoKV owns which immutable revisions are visible and retained by a workspace.

Documentation

• Documentation index
• Product design
• Architecture
• Workbench contract
• Metadata schema
• Object layout
• Path-native metadata comparison
• Workspace acceptance
• Live deployment preflight
• Agent contributor handbook
• Code contract
• PR review checklist
• Source-only Homebrew release

Third-party Listings

NoKV is listed in the LF AI & Data Landscape and Awesome Rust. The DBDB.io profile documents NoKV's earlier Go storage-engine line. Listings are discovery metadata, not foundation-hosted status, integration, deployment, or qualification evidence.

Open-source collaborations. Active: LoopX and OpenViking. Projects initiated: Hermes Agent and heima. These labels describe collaboration stage, not production adoption or completed qualification.

Contributing

Read CONTRIBUTING.md, the code contract, and the PR review checklist before changing package boundaries or durable storage semantics. Every commit requires a DCO Signed-off-by trailer.

Before pushing a substantial change, run:

cargo fmt --all -- --check
cargo clippy --workspace --all-targets -- -D warnings
cargo test --workspace
python3 scripts/workbench/workbench_contract_test.py
git diff --check

Crates

License

Apache-2.0. See LICENSE.