MetaScreener

Open-source multi-LLM ensemble for systematic review screening

Overview

MetaScreener automates the screening phase of systematic reviews using an ensemble of open-source large language models. Instead of relying on a single model, it runs multiple LLMs in parallel and aggregates their outputs through a calibrated confidence pipeline — producing transparent, reproducible decisions with uncertainty quantification.

Key idea: Upload your search results from PubMed/Scopus/etc., define your review criteria (PICO/PEO/SPIDER), and MetaScreener reads each title & abstract — returning include/exclude decisions with confidence scores. Uncertain cases are routed to human review.

Why MetaScreener?

Architecture

MetaScreener uses a Hierarchical Consensus Network (HCN) — a 4-layer screening pipeline:

                        ┌──────────────────────────────────┐
    Upload              │         Layer 1: Inference        │
  .ris/.bib ──────────▶ │  4+ LLMs run in parallel via API  │
  .csv/.xlsx            │  (DeepSeek, Qwen, Llama, Kimi…)  │
                        └──────────────┬───────────────────┘
                                       ▼
                        ┌──────────────────────────────────┐
    PICO/PEO/           │       Layer 2: Rule Engine        │
    SPIDER  ──────────▶ │  Hard rules (auto-exclude)        │
    Criteria            │  Soft rules (score penalties)      │
                        └──────────────┬───────────────────┘
                                       ▼
                        ┌──────────────────────────────────┐
                        │     Layer 3: CCA + ECS            │
                        │  Calibrated Confidence Aggregation │
                        │  Element Consensus Scoring         │
                        └──────────────┬───────────────────┘
                                       ▼
                        ┌──────────────────────────────────┐
                        │    Layer 4: Decision Router        │
                        │  Tier 0 → Hard-rule exclusion      │
                        │  Tier 1 → High-confidence auto     │
                        │  Tier 2 → Moderate auto-decide     │
                        │  Tier 3 → Human review             │
                        └──────────────────────────────────┘

Supported Models

15 open-source LLMs via OpenRouter, organized by capability:

Recommended presets:

Quick Start

Prerequisites

• Python 3.11+
• An API key from OpenRouter (free sign-up, pay-per-use)

Option A: Install with pip

pip install metascreener

Then start the Web UI:

python -m metascreener
# → Open http://localhost:8000

Option B: Docker (no Python needed)

docker pull chaokunhong/metascreener:latest

docker run -p 8000:8000 \
  -e OPENROUTER_API_KEY="sk-or-v1-your-key-here" \
  chaokunhong/metascreener
# → Open http://localhost:8000

Option C: From source (for developers)

Requires uv and Node.js 18+.

git clone https://github.com/ChaokunHong/MetaScreener.git
cd MetaScreener

uv sync --extra dev        # Install Python dependencies
python run.py              # Start FastAPI + Vite dev servers
# → Backend: http://localhost:8000
# → Frontend: http://localhost:5173

Configuration

Set your API key via one of these methods:

1. Web UI — Go to Settings page and paste your key
2. Environment variable:

   export OPENROUTER_API_KEY="sk-or-v1-your-key-here"
   

Web UI Workflow

MetaScreener provides a modern web interface (Vue 3) with a guided step-by-step workflow:

Screening Flow

1. Upload your search export (.ris, .bib, .csv, .xlsx)
2. Define criteria — type your research question and let AI generate PICO criteria, or upload existing criteria
3. Run screening — papers are processed one-by-one with real-time progress
4. Review — each paper shows: decision, confidence score, tier, per-element consensus, model agreement
5. Override — disagree with a decision? Override it and the system learns from your feedback
6. Export — download results as CSV, Excel, JSON, or RIS

How It Works

Layer 1 — Parallel LLM Inference

Each paper is sent to 4+ LLMs simultaneously. Each model returns:

• Decision (include / exclude / uncertain)
• Confidence score (0–1)
• Per-element assessment (population, intervention, comparison, outcome)
• Reasoning chain explaining the decision

Layer 2 — Semantic Rule Engine

Hard rules auto-exclude papers that violate non-negotiable criteria (e.g., wrong language, animal study when humans required). Soft rules apply score penalties for partial matches.

Layer 3 — Calibrated Confidence Aggregation (CCA)

Raw model scores are calibrated using Platt scaling or isotonic regression, then aggregated with tier-weighted blending. The Element Consensus Score (ECS) measures per-element agreement across models — if all models agree that Population matches but disagree on Outcome, this is captured precisely.

Layer 4 — Decision Router

Papers are routed to one of four tiers:

Supported File Formats

Project Structure

MetaScreener/
├── configs/models.yaml            # Model registry + thresholds (single source of truth)
├── src/metascreener/
│   ├── api/                       # FastAPI backend + routes
│   ├── core/                      # Pydantic models, enums, exceptions
│   ├── io/                        # File readers/writers, PDF parser
│   ├── llm/                       # LLM backends, adapters, parallel runner
│   ├── criteria/                  # PICO/PEO/SPIDER criteria generator
│   ├── module1_screening/         # HCN 4-layer screening pipeline
│   │   ├── layer1/                #   Parallel LLM inference
│   │   ├── layer2/                #   Semantic rule engine
│   │   ├── layer3/                #   CCA + ECS aggregation
│   │   └── layer4/                #   Decision router
│   ├── module2_extraction/        # PDF data extraction
│   ├── module3_quality/           # Risk of bias (RoB 2 / ROBINS-I / QUADAS-2)
│   └── evaluation/                # Metrics, calibration, visualization
├── frontend/                      # Vue 3 + TypeScript + Vite
├── tests/                         # Unit + integration tests (all offline)
├── validation/                    # Reproducibility experiments
├── docker/Dockerfile              # Multi-stage build (slim / full)
├── run.py                         # Dev server launcher
└── pyproject.toml

Evaluation Metrics

MetaScreener computes standard systematic review metrics:

• Sensitivity (recall) — proportion of relevant studies correctly identified
• Specificity — proportion of irrelevant studies correctly excluded
• WSS@95 — Work Saved over Sampling at 95% recall
• AUROC — Area Under Receiver Operating Characteristic curve
• Brier Score — calibration quality (how well confidence matches correctness)
• Calibration Curves — visual diagnostic for model probability calibration
• Bootstrap 95% CI — confidence intervals on all metrics

Development

# Install dependencies
uv sync --extra dev

# Run tests (all offline, no API keys needed)
uv run pytest

# Lint
uv run ruff check src/

# Type check
uv run mypy src/

# Dev server (FastAPI + Vite hot reload)
python run.py

Docker Build

# Slim build (daily use)
docker build -f docker/Dockerfile --target slim -t metascreener .

# Full build (reproducibility experiments + visualization extras)
docker build -f docker/Dockerfile --target full -t metascreener:full .

Reproducibility

MetaScreener is designed for TRIPOD-LLM compliant reproducibility:

• temperature = 0.0 on all LLM calls (deterministic output)
• seed = 42 on all stochastic operations
• Full audit trail: every decision logs model outputs, rule violations, confidence scores, and timestamps
• Session persistence: all screening sessions saved as JSON for re-analysis
• Calibration diagnostics: Brier scores, calibration curves, and confidence intervals

Tech Stack

Citation

If you use MetaScreener in your research, please cite:

@software{hong2025metascreener,
  author    = {Hong, Chaokun},
  title     = {{MetaScreener}: Open-Source Multi-LLM Ensemble for Systematic Review Screening},
  year      = {2025},
  url       = {https://github.com/ChaokunHong/MetaScreener},
  license   = {Apache-2.0}
}

License

Apache License 2.0 — free for academic and commercial use.