Agentic Engineering Handbook

The definitive OpenAI, Anthropic, Google, MCP, Harness, Evals, and Production Agent Systems learning roadmap.

If this repository helps you, consider giving it a ⭐

Why This Repository?

The AI industry has entered the Agentic Era. Building production-grade AI systems now requires mastering agents, tool use, MCP, memory, long-running workflows, coding agents, agent harnesses, evals, and safety — but the knowledge is scattered across OpenAI blogs, Anthropic engineering posts, SDK docs, cookbooks, and research papers.

This repository consolidates 215 curated resources into one structured learning roadmap.

The goal: Become a world-class Agentic Engineer.

How To Use This Handbook

Pick the path that matches your starting point:

• New to agents: follow the Learning Roadmap through the core Phases 0–6, then continue to Phase 7 for advanced coursework. Treat each Read First, Then Read, and Build Exercise as a checklist.
• Already building LLM apps: start at Phase 2 or Phase 3, then fill gaps in agent loop, tool calling, evals, and production engineering.
• Trying to build projects: use the phase-level Build Exercise prompts, then branch into Applied Practice Tracks for coding agents, security, code review, or SRE.
• Looking for references: jump to the Full Reading Table. Read P0 first, use P1 for implementation detail, and keep P2 as optional background.

AI Engineering Skills Map

Start with Andrew Ng's The AI Engineering Skills Map (P0) for a broader view of the skills behind this roadmap. It identifies four areas; the connections below are this handbook's interpretation.

For each project, define the user problem and success criteria, choose an appropriate scope, then use implementation and evaluation to test those decisions.

Learning Roadmap

Phase 0 — Agent Loop From Scratch

If you treat Claude Code as a coding CLI, many capabilities can feel like magic: it reads files, runs commands, edits code, delegates work, and stays oriented during complex tasks.

From an engineering perspective, the core is much simpler:

model + tools + one loop.

Understanding that loop makes the rest of the system easier to reason about:

• When the agent should plan first, and when it should act immediately
• Why an explicit todo list reduces drift in longer tasks
• Why subagents improve exploration while protecting the main context
• How skills, MCP, and hooks each add capability around the same core loop

These pages are based on the upstream English Markdown tutorials from shareAI-lab/mini-claude-code, with added Study Notes and inline source code for this handbook.

Supporting files are included in the same folder: requirements.txt, .env.example, v0_bash_agent_mini.py, and skills/.

Next reference: minion.py is a compact, single-file coding agent worth reading after this lab. It shows how an OpenAI-compatible agent loop grows to include tool calls, sessions, resume, approvals, memory, and context compaction.

Phase 1 — Agent Foundations

Build shared vocabulary for workflow vs agent, tool loop, handoff, guardrails.

Key Mental Models

Why agents matter: AI as leverage

What if your impact could grow without every new result requiring another hour of your work? In his Cornell lecture, Hyung Won Chung uses leverage—getting more output from a given input—to explain the opportunity. Drawing on Naval Ravikant's framework, he groups leverage into human labor, capital, and code/media:

AI agents combine human labor and code leverage: they can perform delegated work while running as software that can be replicated. Chung connects this to individuals and small teams doing more, lower barriers to learning, and AI helping synthesize knowledge for scientific progress. The engineering opportunity is to turn that potential into reliable work through tools, state, verification, and recovery.

Start with your own bottleneck: Which recurring task could you delegate, reuse across projects, or make available to more people? Use that task as a candidate for the agent you build in this roadmap.

Source: Hyung Won Chung (OpenAI) — Cornell AI history lecture.

Should I build an agent? (4-question checklist from Barry Zhang's talk - Anthropic)

Think like the agent. Most failures come from designing with a human perspective. Put yourself inside the agent's context window: you only see ~10K–20K tokens (system prompt + tool descriptions + recent observations). Ask: does the agent have enough information to act correctly at each step?

→ Source: How We Build Effective Agents

Read First

Then Read

Decision components: For classification, routing, and rubric-based scoring, see Decision Models — Use, Train & Evaluate in Phase 7. Read the architecture overview and API guide first; return to the training tutorial when you are ready to build your own decision model.

Prompting experiment: Read the provider guides first, including the guidance for your chosen model. Treat prompt repetition as a hypothesis to test: compare the original input with two copies on a fixed dataset, measuring accuracy, input/output tokens, cost, and latency. The paper tested early-2025 models, with stronger gains without reasoning and smaller gains when reasoning was encouraged; long repeated inputs can increase latency.

Make agent outputs easier to understand

As agents take on more execution, understanding and reviewing their results becomes a larger part of your work. Karpathy suggests four output formats you can ask an LLM to create:

These are practical suggestions for choosing a useful output format. A custom page or video can be created for one learning task and discarded afterward. Choose the format that helps you understand and check the result.

Try it: Explain the same agent workflow as plain text, a diagram, and an interactive page. Compare how easily you can find missing steps, unclear permissions, and unsupported claims; check each explanation against the actual workflow.

Source: Andrej Karpathy on understanding LLM outputs.

Build Exercise

Build a customer service/ticket triage agent: router → specialist → evaluator, with all outputs constrained by structured schemas.

Phase 2 — MCP & Tool Ecosystem

Understand MCP server/client, remote vs local, tool loading, approval, connector boundaries.

Read First

Then Read

Event integration note: Study event discovery, subscription creation and refresh, callback verification, and unsubscribe. This guide describes ChatGPT's MCP Events integration: it requires MCP 2.0 and currently supports webhook delivery. Test authorization, duplicate deliveries, restart recovery, and feedback loops; treat event text as data rather than instructions.

Build Exercise

Build a read-only repo/docs MCP server, then create an eval to verify the agent correctly cites documentation.

Phase 3 — Context, Memory & Skills

Learn to control context window, short/long-term memory, skills/plugins, CLAUDE.md/AGENTS.md.

Read First

Reading note: Read The new rules of context engineering after Effective context engineering for AI agents. When upgrading models, use evals to identify outdated instructions that can be removed, keep repository guidance focused on project-specific gotchas, and load detailed skills only when needed. Anthropic's reported removal of over 80% of Claude Code's system prompt without measurable loss on its coding evals is a model- and product-specific result, not a universal reduction target.

Then Read

Skill authoring practice: Read the authoring guide for discovery descriptions, task-appropriate freedom, progressive disclosure, and evaluation-driven iteration. Then inspect Vercel's design.md as a concrete example combining a scoped workflow, design constraints, CSS resources, and visual verification. Its brand preferences and official-authorship assumptions are specific to Vercel; adapt the structure to your own product and identity.

Skills in an ML workflow: Use Hugging Face's skill collection as a practical example of packaging dataset operations, model training, evaluation, and experiment tracking for coding agents. For the exercise below, choose a small dataset task and inspect the corresponding SKILL.md and helper scripts; compare its results and token cost with the same task without a skill.

Instruction and trust reading: Start with the introduction, then consult the Constitution's sections on principals and conversational inputs. Study how Anthropic distinguishes operator instructions, user requests, and tool or document content when handling conflicts. These describe intended Claude behavior; use evaluations to check actual behavior in your application.

Long-context reference: Consult the Claude prompting guide for document placement, XML structure, and grounding answers in source passages; verify model-specific advice with your own evaluations.

Build Exercise

Implement the same task as a Skill/Plugin, then measure accuracy and token cost across three variants: no skill, long prompt, and skill-based.

Phase 4 — Harness & Long-Running Agents

Master agent runtime: event stream, thread, tool execution, state, sandbox, approval, recovery.

Read First

Then Read

Code walkthrough: Revisit minion.py after the harness readings. Trace its tool execution and approval flow, session persistence and resume, and context compaction to see how these runtime responsibilities fit together in a single-file coding agent.

Event-triggered work: See MCP Events in Phase 2 for connecting external updates to agent work through subscriptions and webhooks. Include subscription state and duplicate-event handling in the runtime design.

Runtime implementation: Study OpenShell's sandbox, gateway, and supervisor architecture to see how filesystem, process, and network policies are enforced around an agent. Follow how credentials are attached to approved requests and how policy changes are checked before granting new access.

Filesystem isolation background (P2): Follow the iximiuz Labs tutorial to understand mount namespaces, rootfs and pivot_root, mount propagation, and bind mounts. Use these mechanisms to reason about which files an agent sandbox can access and which paths are shared with its host. The tutorial explains container filesystem mechanics; a complete agent sandbox also needs process, network, and permission controls.

Long-run operating guide: See the Opus 5.5 playbook in Phase 5 for specifying when an agent should continue or stop, keeping progress in a task file, and checking evidence from delegated work.

Build Exercise

Build a mini coding harness: plan file, shell tool, apply patch, test gate, event log, and resume capability.

Phase 5 — Coding & Workspace Agents

Compare Codex vs Claude Code product/SDK forms; learn multi-agent, IDE, workspace collaboration.

Read First

Then Read

Spec-driven practice: Follow Paul Everitt's course to define a project constitution (mission, tech stack, and roadmap), then plan, implement, and validate one feature before replanning. Apply the workflow to a new or existing codebase, and capture repeatable steps in an agent skill.

Coding-agent course: Use CS146S (Fall 2026) alongside this phase for a structured path from coding-agent internals and context engineering to repository setup, code review, security, background agents, and team workflows. Consult the syllabus for available slides, example code, and selected lecture transcripts; later materials are released during the semester.

Model-specific playbook: Use the Opus 5.5 article for completion criteria, stop conditions, persistent task checklists, evidence checks for delegated work, and reporting what could not be confirmed. For general prompting techniques, use the Claude prompting reference in Phase 1. Re-check model-specific recommendations on your own tasks.

Team lifecycle practice: Read the SDLC playbook after the spec-driven course to connect Plan, Design, Build, Test, Deploy, and Maintain. Track intent, specifications, implementation plans, code and tests, review findings, and incident records as linked artifacts. Define the authoritative record for each artifact and the human review gates; feed production findings back into the next change.

Build Exercise

Run both OpenAI/Codex and Claude Code style workflows on the same repo: issue → plan → patch → tests → PR summary.

Phase 6 — Evals, Safety & Production

Build pre/post-launch eval loop, trace loop, safety boundaries, permissions, regression monitoring.

Read First

Security reading note: Focus on §3 for isolation patterns and §4.3/§4.10 for email/calendar and software-engineering agents. Guarantees depend on the threat model and enforced system constraints: fixing a plan can protect control flow while still allowing untrusted data to corrupt output content.

Then Read

Auto mode reading note: Study the separation of input screening and action approval, the classifier's restricted view of user messages and tool calls, and delegation checks. The article reports 0.4% false positives on real traffic and 17% false negatives on 52 real overeager-action examples: an engineering tradeoff, not a safety guarantee.

Paired learning path: Read the AI Engineering Loop to connect production tracing and monitoring with datasets, experiments, and evaluation. Then complete the workshop using its reference app and checkpoint tags: add tracing, version prompts, monitor behavior, build a dataset, and compare a change against a baseline. Follow each lesson's checkpoint and verification steps.

Expert-feedback case study: Study Uber’s Redlining Agent for capturing experts’ final decisions, edited text, and reasoning; retrieving time-weighted examples; separating fixed policies from negotiable preferences; and reviewing suggestions inside Word. Its “self-learning” updates retrieved few-shot context rather than model weights. Connect this feedback memory to Phase 3 and examine the retrieval-quality and rule-adherence evaluations. The article describes the first-generation system; modern harnesses, skills, and knowledge graphs are exploration directions.

Behavior and oversight reference: Revisit the paired Constitution readings in Phase 3 for intended behavior around instruction authority and human oversight. Connect those expectations to the permissions, isolation, and safety evaluations you implement here.

Isolation practice: Revisit OpenShell in Phase 4 and define the minimum file, process, and network access needed for one task. Verify that the task succeeds with allowed access, forbidden file and network requests are blocked, and a requested policy expansion goes through the intended review flow. Record the results as regression cases.

Cloud isolation background (P2): Read the Nitro overview first, then the security design whitepaper for dedicated I/O hardware, the hardware root of trust, and a minimized hypervisor. Compare this EC2 infrastructure isolation with OpenShell's agent execution policies. Nitro Enclaves provide an additional isolated environment for sensitive processing; tool permissions, credential management, and application-level safety still need their own controls.

Lifecycle governance reference: Revisit the AI-native SDLC playbook in Phase 5 for continuous evaluation, deployment checks, review records, and turning production incidents into new intent. Use its governance and measurement sections to connect your safety checks to the team's release and maintenance workflow.

Build Exercise

Build a smoke/macro eval suite for your agent: task success rate, tool misuse, prompt injection resistance, latency, cost, and human approval count.

Phase 7 — Advanced Coursework: Agent Training & Search

Extend agent engineering into model training, reinforcement learning, and search.

CMU 11-768 extends the roadmap from evaluating agents to training and search. Its schedule starts with fundamentals, then develops three areas that make it a valuable advanced follow-on:

• Training methods: supervised fine-tuning (SFT), reinforcement learning basics, advanced RL algorithms, and RL systems.
• Search and inference: reranking, critic models, and tree search.
• Integrated practice: assignments progress through Harness → Eval → Training, connecting agent infrastructure and evaluation to learned behavior.

When to study: Phase 6 provides a useful bridge because evaluation supplies feedback for studying training and search methods. This placement is the handbook's recommendation, not an official course prerequisite; the opening lectures can be used earlier as review.

Use both links together: Open each lecture in the schedule for its slides and readings, then watch the matching recording and follow the assignment milestones. Later scheduled topics may not yet have recordings.

Optional — Agent Systems & Inference Optimization (P2)

Harvard CS2680: Modern AI Systems — Agents and System Optimizations (Fall 2026) complements CMU's training and search focus with agent and inference-system optimization. Follow its public slides, readings, notes, and assignments to study task success, cost, and latency across the agent loop and serving stack: batching and scheduling, KV-cache and prefix reuse, routing, quantization, and speculative decoding.

The five assignments progress from observing an agent to building its loop, optimizing token use, serving an open-weight model, and optimizing the full stack. The systems-heavy second half makes this an advanced elective; the course expects CS61 plus at least one graduate-level computer systems course and familiarity with Python/PyTorch. Lectures are not recorded. Materials are released during the semester; self-study requires your own API and compute resources.

Decision Models — Use, Train & Evaluate (P1)

Study these four resources together to move from understanding decision-model architecture to using, training, and evaluating a decision component. This is component-level decision modeling, rather than full agent training.

Start with Clef’s architecture: a prefill-only backbone pass followed by parallel schema-choice scoring, with a decision head and LoRA adapters trained using cross-entropy, Brier loss, and a secondary RLCD objective. Treat the reported benchmark and latency results as Cloudflare’s measurements to validate on your own workload. Next, study predicates, fixed-choice classification, and ordered scoring in the API guide, including how they differ from structured generation and tool calls. Then follow the Unsloth tutorial and notebook through dataset preparation, 4-bit LoRA and decision-head training, probability calibration, separate test-set evaluation, and model saving. The Decisions API is currently in public beta; check the guide for supported models and availability.

Practice: Compare a hosted decision API and a locally trained decision model on a labeled request-routing task. Measure accuracy, probability calibration, latency, and cost; choose review thresholds based on false-positive and false-negative costs, and handle low confidence, out-of-scope inputs, and refusals. Keep the final test set separate from training, calibration, and threshold selection. Calibration on one dataset does not guarantee reliable probabilities on your own traffic. Unlike the feedback retrieval in Uber's Redlining Agent in Phase 6, this Unsloth workflow trains model parameters and a decision head.

Then Read — Post-Training Case Study (P1)

Rufus-Air: An Open LLM Post-Training Recipe (Amazon) connects the course's training topics to an end-to-end recipe on GLM-4.5-Air-Base: SFT → Reasoning RL → Coding RL → Instruction-Following RL → General Agent → Coding Agent → Search Agent → RLHF.

Focus on §2 for stage ordering and reward reliability, §3.5–3.7 for agent training in tool-use, coding, and search environments, and §4 for multi-turn rollouts, sandbox reliability, and consistent training/inference formats. Some conclusions come from engineering experience rather than full ablations; treat the stage order as a reference recipe to evaluate for your own setting.

Further Reading — AI Research Automation (P2)

When AI builds itself examines evidence that AI is accelerating AI development, remaining gaps in research judgment, and possible futures. Read it as background on research automation and governance: full recursive self-improvement is a prospective scenario, not an achieved capability.

Applied Practice Tracks

Use these tracks after the core roadmap when you want to practice agentic engineering in real engineering workflows.

Full Reading Table

Priority guide: P0 = must-read (architectural/conceptual), P1 = highly useful (implementation detail), P2 = optional context (background/releases).

Who Is This For?

• AI Engineers
• Agent Engineers
• LLM Engineers
• Platform Engineers
• Research Engineers
• AI Startup Founders

Contributing

Contributions are welcome. If you find:

• New OpenAI resources
• New Anthropic resources
• MCP updates
• Agent evaluation frameworks
• Production engineering articles

Please open a pull request.

Vision

The goal of this project is to become the System Design Primer for Agentic Engineering.

If you're serious about building production AI agents, start here.

Star History

License

MIT