PaperGuru-Benchmark

AutoTrustAI/PaperGuru-Benchmark is ranked 10,679 among these 8 peers; study8677/repobrain is ranked 10,678.

**PaperGuru** is the first system designed from a 4-axiom formalisation of *Lifecycle-Aware Memory* (LAM), and on the two most rigorous published benchmarks it delivers state-of-the-art from a single algorithmic mechanism. 21%** | | **PaperBench** | Papers above 41% human ML-PhD bar | **20 / 23** | 4 / 23 | **+16 papers** | | **SurveyBench** (Yan et al. 06%** | | **SurveyBench** | Composite richness (figures · tables · code) | **43. The difference is the memory primitive underneath. PaperGuru is a memory architecture for long-horizon LLM agents. It

is **not** another RAG library and **not** another agent framework. The four LAM axioms (formalised in §3 of the paper): 1. **Structural multi-hop relevance** · the right evidence is two citations away, not one cosine-similarity hop. **Bounded query cost under unbounded archive growth** · the archive grows every day; routing cost cannot grow with it. **Provenance-grounded composition** · every claim in the agent's output must trace back to a verifiable artifact in memory. The dominant unit of LLM deployment is

no longer the single-prompt completion; it is the **long-horizon agentic system**: a multi-day software-engineering session that touches hundreds of files, a literature-grounded research assistant that drafts a 200K-token survey from a citation graph spanning ten years, a paper-to-code reproduction agent that turns a paper PDF into a runnable submission tree, a clinical-evidence agent that reads a decade of trial records before recommending treatment. It is set by what the system *remembers

* between turns and *retrieves* from outside the working set. ** Flat retrieval breaks the moment a paper is revised; agent-specific memory hacks (MemGPT tiers, Ebbinghaus forgetting, knowledge-graph wrappers) each handle one or two of the four axioms but never all four. CCM is the first design we know of that satisfies all four jointly without per-task hand-tuning. The Reason stage is where the Compose / Critique / Mutate cycle runs. Three sub-blocks of the Reason stage cycle through

Compose / Critique / Mutate; verification ticks resolve as evidence cards land. The official human-expert baseline is **41% over a 48-hour ML-PhD budget**. Aggregate PaperGuru reaches a **per-paper mean of 66. 05% across all 23 papers**, beating every published baseline and clearing the human-expert bar by **+25 points**. Per-paper For the 20 papers that have a published baseline: **19 / 20** papers improve over the strongest baseline Mean lift: **+30. 47%, where the cited PINN baseline already

used hand-tuned domain priors) Distribution of lifts Three additional papers ( , , ) have no published baseline; PaperGuru scores **95. 📂 **All 23 reproduction submissions are in . ** Aggregate scores are in and the per-paper comparison report is in . PaperGuru is evaluated under the official judge with all dimensions normalised to [0, 100%]. Content quality (5-axis radar) PaperGuru scores **94. 06%** absolute lift over the strongest baseline (AutoSurvey at 80. 06%** | Composite richness (the structural advantage) Richness measures whether the generated survey

contains *evidence-grounded* artifacts: figures, tables, executable code blocks, and resolved citations. 📂 **All 20 generated surveys are in ** in three