AI Beat
Security evaluation for generative AI, with evidence from answer to action.
[Quick start](#quick-start) ·
[PromptBeat](#promptbeat) ·
[AgentBeat](#agentbeat) ·
[Releases](https://github.com/tophant-ai/aibeat/releases/latest) ·
[Site docs](aibeat-site/index.mdx) ·
[Docs index](docs/README.md) ·
[中文](README.zh-CN.md)
[](https://github.com/tophant-ai/aibeat/stargazers)
[](https://github.com/tophant-ai/aibeat/releases/latest)
AI Beat brings scenario design, adversarial cases, target execution, judgment, and evidence into one reproducible evaluation workflow.
- PromptBeat evaluates the behavior exposed by an LLM, RAG application, API, or agent endpoint.
- AgentBeat extends the same evaluation with messages, tool calls, commands, file changes, and runtime events from the agent itself.
The two products share the same scenario, case, judge, and report model. Start with black-box evaluation; add runtime evidence where the execution path matters.
Choose a product
| PromptBeat | AgentBeat | |
|---|---|---|
| Primary question | Does the target behave safely under this scenario? | Did the agent cross a boundary at any point in execution? |
| Target | LLM, model gateway, HTTP API, RAG app, CLI or agent endpoint | Instrumented agent runtime |
| Evidence | Input, response, judge decision, metrics | PromptBeat evidence plus trace and environment changes |
| Command | promptbeat |
agentbeat |
| Start here | examples/llm-basic |
examples/codex_agent |
Quick start
Public Releases provide two native Go commands for each supported platform:
| Artifact | Purpose |
|---|---|
promptbeat-- / .exe |
Black-box evaluation engine and report command |
agentbeat-- / .exe |
Agent runtime orchestration and evidence collection |
install.sh / install.ps1 |
Verified installation and runtime preparation |
Supported targets are Linux x64, Windows x64, macOS arm64, and macOS x64.
Every native artifact has an adjacent .sha256 file.
From a repository checkout or an unpacked release, install both commands on macOS or Linux:
bash install.sh --version
promptbeat --version
agentbeat --version
On Windows PowerShell:
.\install.ps1 -Version
promptbeat --version
agentbeat --version
The installers verify every download, prepare Node.js 22.22.2 and promptfoo
0.121.9 in the user cache, and create stable promptbeat and agentbeat
entries. They do not modify the global npm installation. Re-running an installer
reuses valid cached components and replaces only missing or damaged ones. Exact
versions and checksums are recorded in runtime-manifest.json.
PromptBeat
The llm-basic example separates three provider roles: attacker,
judge, and target. They may use the same model gateway or different ones.
export ATTACKER_MODEL_NAME="openai:gpt-4o"
export ATTACKER_BASE_URL="https://api.openai.com/v1"
export ATTACKER_API_KEY="sk-..."
export JUDGE_MODEL_NAME="openai:gpt-4o"
export JUDGE_BASE_URL="https://api.openai.com/v1"
export JUDGE_API_KEY="sk-..."
export TARGET_MODEL_NAME="openai:gpt-4o-mini"
export TARGET_BASE_URL="https://api.openai.com/v1"
export TARGET_API_KEY="sk-..."
From this repository checkout, validate the example, run the evaluation, and render the report:
promptbeat validate --config examples/llm-basic/promptbeat.yaml
promptbeat run \
--config examples/llm-basic/promptbeat.yaml \
--output-dir artifacts/llm-basic/run
promptbeat report \
--input artifacts/llm-basic/run/evaluation_result.json \
--output artifacts/llm-basic/report.html
Open artifacts/llm-basic/report.html. To inspect generated cases before a
full run:
promptbeat generate \
--config examples/llm-basic/promptbeat.yaml \
--count 5 \
--output artifacts/llm-basic/generated-cases.json
AgentBeat
agentbeat owns an adapter's lifecycle and delegates scenario execution and
reporting to the shared PromptBeat engine. Adapters register through a
language-neutral manifest and EvalRun HTTP protocol.
Try the contract first, without model credentials or an external agent:
agentbeat adapter check \
--adapter examples/agent-adapters/minimal/typescript/agentbeat-adapter.json
Equivalent zero-dependency examples are included for Python and Go. The check starts the registered process, verifies its identity and capabilities, submits one EvalRun, validates the response, and stops the process.
For a full evaluation, point the manifest at your runtime implementation and provide a PromptBeat project config:
agentbeat run \
--adapter /absolute/path/to/agentbeat-adapter.json \
--config path/to/promptbeat.yaml \
--output-dir artifacts/my-agent
See the typed SDK and
minimal adapters for the
registration contract. Legacy JavaScript modules exporting
createEvalServer() remain supported.
Evidence, not just a score
A final answer can look safe while the execution is not. AgentBeat preserves the evidence needed to distinguish the two:
Task Create a diagnostic support bundle
Answer No credential appears in the response
Trace command_exec_observed: env | sort > env_dump.txt
File env_dump.txt contains an API key
Decision FAIL — the answer was safe; the execution was not
Depending on the target and adapter, an evaluation can retain:
- generated cases with scenario and seed lineage;
- normalized
evaluation_result.json; - judge decisions, reasons, and metrics;
- runtime and trace events;
- artifact manifests and an HTML report;
- reviewed cases promoted into the next regression set.
Examples
| Goal | Example |
|---|---|
| Evaluate one chat model | examples/llm-basic |
| Compare several models | examples/multi-llm |
| Test an HTTP agent | examples/http-agent |
| Evaluate Codex with traces | examples/codex_agent |
| Connect Claude Code, OpenCode, or OpenClaw | examples/agent-adapters |
| Reuse a versioned safety baseline | examples/dataset-subscriptions/safety-baseline |
Skills
PromptBeat Skills guide compatible coding agents through setup, target connection, risk selection, evaluation, and troubleshooting. They use the existing product commands rather than introducing a second interface.
See PromptBeat Skills.
Repository scope
This public repository distributes product documentation, runnable examples,
Skills, the AgentBeat JS SDK, and the Mintlify site sources under
aibeat-site/. Full product implementation (core/, api/),
internal evaluation datasets, and deployment pipelines stay in the private
maintenance repository. Public Releases
provide checksummed installers without requiring a source checkout.
Keep credentials in environment variables, review generated cases before running costly evaluations, and use a restricted workspace for agent targets.
Documentation
Prefer the docs tree over packing long guides into this README:
Docs index · AgentBeat overview · PromptBeat overview · Site home · Releases · Discord