Skip to navigation

Webpage and PDF Extraction

Fetch any webpage or PDF and parse it into clean Markdown or HTML for LLM ingestion.
View as MarkdownOpen in Claude

The Extract API fetches any webpage or PDF and parses it into clean Markdown or HTML, plus page metadata, without a headless browser or an HTML parser on your side. PDF tables keep their rows and columns, so filings and reports arrive ready for an LLM.


What You’ll Build

A one-call extractor that fetches a URL and parses it into clean Markdown—title, body, and page metadata in a single response. full_page.extraction_formats picks Markdown, HTML, or both, and timeout caps the whole request.


Try It Live

Run a live Extract API request from this page. Open the Try It panel below, paste your API key, drop in a URL, and send it against the live endpoint.

POST
/v1/extract
curl -X POST https://ydc-index.io/v1/extract \
-H "X-API-Key: <apiKey>" \
-H "Content-Type: application/json" \
-d '{
"url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
"full_page": {
"extraction_formats": [
"markdown",
"html"
]
}
}'
Response
{
"url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
"extraction_source": "blend",
"title": "Retrieval-augmented generation - Wikipedia",
"contents": {
"full_page": {
"markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources...",
"html": "<h1>Retrieval-augmented generation</h1>\n<p><b>Retrieval-augmented generation</b> (<b>RAG</b>) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources...</p>"
}
},
"metadata": {
"site_name": "Wikipedia",
"favicon_url": "https://api.ydc-index.io/favicon?domain=en.wikipedia.org&size=128"
}
}

Prerequisites

Install the Python SDK. you.extract() needs version 3.7.0 or later.

pip install -U youdotcom # Python ≥ 3.10

Walkthrough

extract.py
"""Extract—fetch clean Markdown from a webpage or PDF via the You.com Extract API."""
import sys
from youdotcom import You
# take URL from command line, or use a default
url = sys.argv[1] if len(sys.argv) > 1 else "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"
with You() as you:
page = you.extract(
url=url,
full_page={"extraction_formats": ["markdown"]},
timeout=30,
)
# an empty object means nothing was extracted
if not page.contents:
sys.exit(f"Nothing extracted from {url}")
# print the title and the first 500 chars of the markdown body
print(page.title)
print(page.url)
print()
print(page.contents.full_page.markdown[:500] + "...")
export YDC_API_KEY="your-api-key-here"
python extract.py "https://en.wikipedia.org/wiki/Retrieval-augmented_generation"

Example Output

# Retrieval-augmented generation
Retrieval-augmented generation (RAG) is a technique that grants generative
artificial intelligence models information retrieval capabilities. It modifies
interactions with a large language model so that the model responds to user
queries with reference to a specified set of documents…
(Returned alongside title, url, and metadata.site_name = "Wikipedia". Full
Markdown body is ~12,000 chars.)

Pass a PDF URL to the same script to extract a document. Add "parse_effort": "lite" to the request for text-based PDFs to skip the per-page charge that the default standard adds.


Next Steps


Resources