mdream

mdream

☁️ Convert HTML to Markdown for LLMs, with Rust and JavaScript engines and streaming support.

[!TIP] 🎉 Upgrading to Mdream v2? Follow Migrating to v2.

Made possible by my Sponsor Program 💖
Follow me @harlan_zw 🐦 • Join Discord for help

Features

• ☁️ Powering Cloudflare Browser Run's /markdown and /crawl endpoints.
• 🧠 Token reduction: Up to 92% fewer tokens than raw HTML in the measured pages.
• 🚀 Conversion speed: Compare Rust and JavaScript results in the benchmarks.
• 🔍 Markdown output: GitHub Flavored Markdown with frontmatter, lists, tables, and code blocks.
• 🌊 Streaming: Convert HTML as input arrives and join the output to match one-shot conversion.
• ⚡ Bundles: The JS engine lets you import formats and plugins separately.
• ⚙️ Integrations: CLI Crawler, Docker, GitHub Actions, and Vite.

What is Mdream?

Mdream converts HTML to Markdown for LLM input. The benchmarks compare conversion speed and token counts with other libraries.

Use the integration packages to generate Markdown pages and llms.txt files from your site.

Mdream Packages

Choose a package for your runtime or build tool:

Package	Description
mdream	Rust NAPI engine + WASM for edge. Performance-first, declarative config. Includes CLI.
@mdream/js	Pure JS engine. Tree-shakable formats and plugins. Zero native dependencies.
@mdream/crawl	Site-wide crawler to generate llms.txt artifacts from entire websites
Docker	Pre-built Docker images: a tiny core HTML-to-Markdown converter and a crawl image with Playwright Chrome
@mdream/vite	Generate automatic .md for your own Vite sites
@mdream/action	Generate .md and llms.txt artifacts from your static .html output
mdream (crate)	Native Rust crate with CLI. Zero dependencies, streaming support. Available on crates.io
Browser CDN	Use mdream directly in browsers via unpkg/jsDelivr without any build step

What can Mdream do?

Pipe any HTML into mdream to get clean, LLM-ready Markdown:

curl -s https://en.wikipedia.org/wiki/Markdown | npx mdream --preset minimal

📥 URL to Markdown

Fetch the Markdown Wikipedia page, then convert it with the minimal preset:

curl -s https://en.wikipedia.org/wiki/Markdown \
 | npx mdream --origin https://en.wikipedia.org --preset minimal \
  | tee streaming.md

The --origin flag resolves relative image and link paths.

Want to make it look nice? Use glow.

curl -s https://en.wikipedia.org/wiki/Markdown \
 | npx mdream --origin https://en.wikipedia.org --preset minimal \
   | glow

📄 Local HTML to Markdown

Convert a local HTML file. The tee command saves the Markdown and displays it in the terminal.

cat index.html \
 | npx mdream --preset minimal \
  | tee streaming.md

Want to make it look nice? Use glow.

cat index.html \
 | npx mdream --preset minimal \
  | glow

🧠 Feed Any Website to an LLM

Pipe web content straight into Claude, GPT, or any LLM CLI:

# Single page → Claude
curl -s https://react.dev/learn | npx mdream --origin https://react.dev --preset minimal \
  | claude -p "explain the key concepts on this page"

# Crawl entire docs → summarize
npx @mdream/crawl "https://nuxt.com/docs/getting-started/**"
cat output/llms-full.txt | claude -p "write a getting started guide from these docs"

# Compare two frameworks
diff <(curl -s https://vuejs.org/guide/introduction | npx mdream --preset minimal) \
     <(curl -s https://react.dev/learn | npx mdream --preset minimal) \
  | claude -p "compare these two frameworks based on their intro docs"

# JavaScript/SPA sites (React, Vue, Angular)
npm install @mdream/crawl crawlee playwright
npx playwright install chromium
npx @mdream/crawl https://spa-site.com --driver playwright
cat output/llms-full.txt | claude -p "what features does this app have"

🌐 Make Your Site AI-Discoverable

Generate llms.txt to help AI tools understand your site:

# Static sites
npx @mdream/crawl https://yoursite.com

# JavaScript/SPA sites (React, Vue, Angular)
npm install @mdream/crawl crawlee playwright
npx playwright install chromium
npx @mdream/crawl https://spa-site.com --driver playwright

Outputs:

• output/llms.txt - Optimized for LLM consumption
• output/llms-full.txt - Complete content with metadata
• output/.md - Individual Markdown files, preserving each URL path

🗄️ Build RAG Systems from Websites

Crawl websites and generate embeddings for vector databases. Install the packages and configure an AI Gateway API key:

pnpm add @mdream/crawl @mdream/js ai
export AI_GATEWAY_API_KEY="your-api-key"

import { crawlAndGenerate } from '@mdream/crawl'
import { withMinimalPreset } from '@mdream/js/preset/minimal'
import { htmlToMarkdownSplitChunks } from '@mdream/js/splitter'
import { embed } from 'ai'

const embeddings: { url: string, title: string, content: string, embedding: number[] }[] = []

await crawlAndGenerate({
  urls: ['https://example.com'],
  output: './output',
  hooks: {
    'crawl:page': async ({ url, html, title, origin }) => {
      const chunks = htmlToMarkdownSplitChunks(html, withMinimalPreset({
        chunkSize: 1000,
        chunkOverlap: 200,
        origin,
      }))

      for (const chunk of chunks) {
        const { embedding } = await embed({ model: 'openai/text-embedding-3-small', value: chunk.content })
        embeddings.push({ url, title, content: chunk.content, embedding })
      }
    },
  },
})

// Save to vector database: await saveToVectorDB(embeddings)

✂️ Extract Specific Content from Pages

Pull headers, images, or other elements during conversion:

import { htmlToMarkdown } from 'mdream'

const headers = []
const images = []

htmlToMarkdown(html, {
  extraction: {
    'h1, h2, h3': el => headers.push(el.textContent),
    'img[src]': el => images.push({ src: el.attributes.src, alt: el.attributes.alt }),
  },
})

⚡ Optimize Token Usage With Clean Mode

Pass clean: true to remove empty links, tracking parameters, and other unwanted output. The minimal preset enables all cleanup rules by default.

import { htmlToMarkdown } from 'mdream'

// All clean features enabled
htmlToMarkdown(html, { clean: true })

// Or selective features
htmlToMarkdown(html, {
  clean: {
    emptyLinks: true, // Strip #, javascript: links
    emptyLinkText: true, // Drop [](url) links with no text
    emptyImages: true, // Strip ![](url) with no alt text
    redundantLinks: true, // [url](url) → url
    selfLinkHeadings: true, // ## [Title](#title) → ## Title
    fragments: true, // Strip broken #anchor links
    urls: true, // Strip utm_*, fbclid tracking params
  }
})

Mdream Usage

Installation

pnpm add mdream

[!TIP] Using an AI agent? Get the mdream Skill on skilld.dev/gh/harlan-zw/mdream.

Bundler Compatibility

In Node.js, mdream loads native bindings that must remain outside the bundle. If your bundler fails to resolve mdream, mark it as external:

Next.js / Turbopack:

// next.config.js
const nextConfig = {
  serverExternalPackages: ['mdream'],
}

Webpack / other bundlers:

externals: ['mdream']

[!TIP] @mdream/js has zero native dependencies and works with all bundlers without configuration.

[!TIP] Using Vite? @mdream/vite handles this automatically.

Basic Usage

import { htmlToMarkdown } from 'mdream'

// Rust NAPI engine in Node.js, WASM in edge runtimes
const markdown = htmlToMarkdown('
Hello World
')
console.log(markdown) // # Hello World

const text = htmlToMarkdown('
Hello World
', {
  format: 'text',
})
console.log(text) // Hello World

const html = htmlToMarkdown('
Hello World
', {
  format: 'html',
})
console.log(html) // 
Hello World

import { streamHtmlToMarkdown } from 'mdream'

const response = await fetch('https://en.wikipedia.org/wiki/Markdown')
for await (const chunk of streamHtmlToMarkdown(response.body, {
  origin: 'https://en.wikipedia.org',
  minimal: true,
})) {
  process.stdout.write(chunk)
}

In a browser bundle, import htmlToMarkdown from mdream/browser and await it. The result is a string.

See the mdream docs for complete details.

Migrating from v1

Use the guide for each package:

• mdream: browser imports and returns, Web Workers, minimal filtering, and page titles.
• @mdream/js: format imports, plugin arrays, cleanup, and callback changes.
• Rust crate: renamed options, private parser types, and splitter heading levels.
• @mdream/crawl: renamed options, shared defaults, and the mdream-crawl command.

The JS guide also covers upgrades from 2.0.0-beta.1. That release used a different plugin API and format option.

Mdream Crawl

Need something that works in the browser or an edge runtime? Use Mdream.

The @mdream/crawl package fetches site pages and converts them with mdream.

• llms.txt: An index of crawled pages with titles, descriptions, and links.
• llms-full.txt: The full Markdown content of the crawled pages.
• Individual Markdown files: One file per page, with paths that mirror the page URLs.

Usage

# Interactive
npx @mdream/crawl
# Simple
npx @mdream/crawl https://harlanzw.com
# Glob patterns
npx @mdream/crawl "https://nuxt.com/docs/getting-started/**"
# Get help
npx @mdream/crawl -h

Docker

Two images for two jobs:

# core — convert HTML to Markdown (native Rust binary, ~600KB, no Node)
curl -s https://example.com | docker run -i --rm harlanzw/mdream:core --origin https://example.com

# crawl — crawl a site / generate llms.txt (Playwright Chrome included)
docker run harlanzw/mdream:crawl "https://site.com/docs/**"
docker run harlanzw/mdream:crawl spa-site.com --driver playwright

Available Images (Docker Hub harlanzw/mdream, also on ghcr.io/harlan-zw/mdream):

• :core - v2 native Rust HTML-to-Markdown converter (stdin → stdout)
• :crawl - v2 site crawler with Playwright Chrome
• :latest - alias of :crawl

See DOCKER.md for complete usage, configuration, and building instructions.

GitHub Actions Integration

Installation

pnpm add @mdream/action

See the GitHub Actions README for usage and configuration.

Vite Integration

Installation

pnpm install @mdream/vite

See the Vite README for usage and configuration.

Nuxt Integration

Installation

pnpm add @mdream/nuxt

See the Nuxt Module README for usage and configuration.

Browser CDN Usage

Load mdream from a CDN without a build step. The script includes the WASM binary and initializes it on load. Like mdream/browser, htmlToMarkdown() returns Promise:

CDN Options:

• unpkg: https://unpkg.com/mdream/dist/iife.js
• jsDelivr: https://cdn.jsdelivr.net/npm/mdream/dist/iife.js

Benchmarks

JavaScript (Node.js)

Pure JS comparison. mdream uses no plugins, Turndown uses GFM plugin for equivalent table/strikethrough support.

Rust (native, release + LTO)

All crates compiled with opt-level=3, LTO, and single codegen unit.

Rust NAPI (Node.js bindings)

For Node.js apps that need native speed. Includes N-API overhead.

CLI (cross-language, includes process startup)

End-to-end cat file | tool > /dev/null via hyperfine. Includes process startup overhead (~20ms for Node.js, ~1ms for Go/Rust).

In these results, mdream's Rust CLI is 2.6-7.5x faster than Go html2markdown. On the 1.8 MB file, the Node.js CLI is 2.2x faster, including its startup time. For conversion speed without startup time, see the JS and Rust tables above.

Streaming

Both mdream engines accept HTML streams and yield Markdown as conversion progresses. Join the output pieces to get the same result as one-shot conversion.

Token Efficiency

With minimal: true, mdream produces up to 92% fewer tokens than raw HTML in these measured pages.

Benchmarks run on real-world HTML using Vitest bench. See full methodology and reproduction steps.

Migrating to v2

Using a coding agent? Give it the v1 to v2 migration prompt. The prompt covers package detection, code changes, and verification.

These docs cover v2. For an earlier release, read the docs at its release tag. The migration guides also cover upgrades from an earlier v2 beta.

Before deploying v2, convert sample pages with your application's options and compare the output. If you use the JS splitter, rebuild stored chunks and embeddings before your application reads them. For your first v2 crawl, set depth, maxPages, and artifacts to limit pages and output files.

Credits

• ultrahtml: HTML parsing inspiration

License

Licensed under the MIT license.