jamesturk/scrapeghost 排名第 10,316，此处有 8 个同类条目；6551Team/opentwitter-mcp 排名第 10,321。

scrapeghost **This project from 2023 is no longer maintained or recommended, the author has no interest in working with commercial LLMs. ** **was** an experimental library for scraping websites using OpenAI's GPT. Source: Documentation: Issues: **Use at your own risk. This library makes considerably expensive calls ($0. 36 for a GPT-4 call on a moderately sized page. ) Cost estimates are based on the OpenAI pricing page and not guaranteed to be accurate. ** Features The purpose of this library is to

provide a convenient interface for exploring web scraping with GPT. While the bulk of the work is done by the GPT model, provides a number of features to make it easier to use. **Python-based schema definition** - Define the shape of the data you want to extract as any Python object, with as much or little detail as you want. **Preprocessing** **HTML cleaning** - Remove unnecessary HTML to reduce the size and cost of API requests. **CSS and XPath selectors** - Pre

-filter HTML by writing a single CSS or XPath selector. **Auto-splitting** - Optionally split the HTML into multiple calls to the model, allowing for larger pages to be scraped. **Postprocessing** **JSON validation** - Ensure that the response is valid JSON. (With the option to kick it back to GPT for fixes if it's not. ) **Schema validation** - Go a step further, use a schema to validate the response. **Hallucination check** - Does the data in the response truly exist on the page? **Cost

Controls** Scrapers keep running totals of how many tokens have been sent and received, so costs can be tracked. 5-Turbo by default, fall back to GPT-4 if needed. ) Allows setting a budget and stops the scraper if the budget is exceeded.