Retrieve page content

Overview

By default, search results include snippets—100–200 words of extracted text per result. Pass the extraction parameter to ask for richer content: full-page Markdown and HTML, or query-relevant highlights.

This unlocks:

• Deep RAG with full document context
• Knowledge base construction from live web data
• Comprehensive content synthesis across sources
• Full article bodies for news results

Two modes are available. Pick the one that matches the latency and token budget of your pipeline.

If you want query-ranked excerpts rather than whole pages, use extraction_mode: "highlights". The two modes are mutually exclusive—pick one per request.

How It Works

Add the extraction object to a POST /v1/search request. The API fetches each result in real time and attaches a contents object to it. The crawl_timeout parameter is a sibling of extraction at the top level of the request body.

When extraction_mode: "highlights", the server rejects crawl_timeout (it only applies to full_page). The Python SDK strips it automatically with a warning before the request goes out, so you can ignore the constraint unless you call the API directly.

Full-page extraction crawls every web and news result in the response. There is no per-section switch. Control the volume with count. With the default count=10, a call returns up to 10 web + 10 news pages.

Get Highlights

Set extraction_mode to highlights. Each result gains a contents.highlights array of query-relevant excerpts. Snippets are omitted in this mode—the search returns the ranked highlights instead.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionMode
with You() as you:
  res = you.search(
    query="transformer architecture explained",
    count=5,
    extraction=Extraction(extraction_mode=ExtractionMode.HIGHLIGHTS),
  )
  if res.results and res.results.web:
    for result in res.results.web:
      highlights = result.contents.highlights if result.contents else None
      print(f"{result.title}")
      print(f"  URL: {result.url}")
      if highlights:
        print(f"  Highlights: {len(highlights)} excerpts")
        for h in highlights[:3]:
          print(f"    - {h[:140]}…")
      else:
        print("  (No highlights returned)\n")

Cap Highlight Tokens

By default, highlights for a 10-result search add up to roughly 6,000–7,500 tokens. To keep a response small enough for a latency-sensitive agent, set highlights.total_max_tokens. The budget covers every web result in the response combined, and accepts values from 1024 to 8192.

The budget is shared across results, so count decides how it is split. With total_max_tokens: 1024 and count: 10, a response typically carries five to 10 results with short highlights. With count: 3, it carries three results with longer ones.

curl -X POST https://ydc-index.io/v1/search \
  -H "X-API-Key: $YDC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "AAPL stock price today",
    "count": 3,
    "extraction": {
      "extraction_mode": "highlights",
      "highlights": {
        "total_max_tokens": 1024
      }
    }
  }'

The cap is approximate, so count tokens with your own tokenizer if your prompt has a hard limit. A value outside 1024–8192 returns a 422.

Get Full Page Content

Set extraction_mode to full_page. Each successfully crawled result gains a contents.markdown (or contents.html) field.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode
with You() as you:
  res = you.search(
    query="transformer architecture explained",
    count=5,
    extraction=Extraction(
      extraction_mode=ExtractionMode.FULL_PAGE,
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
    ),
  )
  if res.results and res.results.web:
    for result in res.results.web:
      print(f"{result.title}")
      print(f"  URL: {result.url}")
      if result.contents and result.contents.markdown:
        print(f"  Content ({len(result.contents.markdown)} chars)")
        print(f"  Preview: {result.contents.markdown[:200]}…\n")
      else:
        print("  (No content retrieved)\n")

Choose a Content Source

extraction_source decides whether full-page content comes from the You.com cache, a live crawl, or a mix of the two. Leave it unset to get blend, which is the right default for most pipelines.

cache is the fastest of the three because it never waits on a crawl, and it trades coverage for that speed—expect some results to come back without a contents object. fetch is the slowest and the freshest.

Only pages crawled live are billed, at $1.00 per 1,000 pages. Cached content carries no extraction charge, so cache costs nothing beyond the base call price and blend bills only the pages it crawls live. See Billing for worked examples.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode, ExtractionSource
with You() as you:
  res = you.search(
    query="nvidia earnings",
    count=5,
    extraction=Extraction(
      extraction_mode=ExtractionMode.FULL_PAGE,
      extraction_source=ExtractionSource.FETCH,
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
    ),
  )

Coding Agents That Must Not Live-Crawl

Use extraction_mode: "highlights" when the agent only needs query-relevant excerpts. Highlights return contents.highlights without a full-page crawl, and extraction_source applies only to full_page.

For full-page content with no outbound fetch, set extraction_mode to "full_page" and extraction_source to "cache". Results with nothing cached omit contents.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode, ExtractionSource
with You() as you:
  res = you.search(
    query="Python logging.Handler.emit",
    count=5,
    extraction=Extraction(
      extraction_mode=ExtractionMode.FULL_PAGE,
      extraction_source=ExtractionSource.CACHE,
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
    ),
  )

The default blend crawls live on a cache miss, so do not use blend or fetch in locked-down agent environments.

When the agent already has URLs and needs a fetch, call the Contents API instead of extraction_source: "fetch" on the Web Search API.

Full Article Bodies for News

Combine full-page extraction with freshness for breaking news pipelines. Web results are crawled in the same call, so read results.news if articles are all you need.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode, Freshness
with You() as you:
  res = you.search(
    query="semiconductor supply chain",
    freshness=Freshness.WEEK,
    count=5,
    extraction=Extraction(
      extraction_mode=ExtractionMode.FULL_PAGE,
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
    ),
  )
  if res.results and res.results.news:
    for article in res.results.news:
      print(f"{article.title}")
      if article.contents and article.contents.markdown:
        print(article.contents.markdown[:400])
      print()

Get Both Markdown and HTML

Pass both formats in extraction_formats to receive contents.markdown and contents.html on every crawled result.

curl -X POST https://ydc-index.io/v1/search \
  -H "X-API-Key: $YDC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "quantum computing breakthroughs",
    "count": 5,
    "extraction": {
      "extraction_mode": "full_page",
      "full_page": {
        "extraction_formats": ["markdown", "html"]
      }
    }
  }'

Control Crawl Timeout

By default the crawler waits up to 10 seconds per page. For latency-sensitive applications, reduce crawl_timeout. For complex or slow-loading pages, increase it (up to 60 seconds). crawl_timeout sits at the top level of the request, not inside extraction, and only applies when extraction_mode: "full_page". Combining it with highlights is invalid.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode
with You() as you:
  # Low-latency pipeline: only wait 3 seconds per page
  res = you.search(
    query="latest Python releases",
    count=5,
    extraction=Extraction(
      extraction_mode=ExtractionMode.FULL_PAGE,
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
    ),
    crawl_timeout=3,
  )
  if res.results and res.results.web:
    for result in res.results.web:
      status = "crawled" if result.contents and result.contents.markdown else "skipped (timeout)"
      print(f"{result.title} — {status}")

HTML vs Markdown

Highlights or Full Page?

A few rules of thumb:

• You want fast, citation-anchored snippets. Use highlights with a moderate count (10–25). Excerpts land in contents.highlights.
• You need the document body to feed a downstream indexer or synthesizer. Use full_page with extraction_formats: ["markdown"].
• You need rendered HTML for scraping or both formats in the same response. Use full_page with extraction_formats: ["html", "markdown"] to receive both contents.markdown and contents.html per result.
• You already know the URLs. Use the Contents API directly. There is no need to search first.

Already Have URLs?

If you have a list of URLs and don’t need to search first, use the Contents API directly. It accepts URLs without a query and returns the same markdown or html content.

Legacy: livecrawl

Before extraction, page content came from the livecrawl parameter. It still works on both GET and POST /v1/search, so existing integrations keep running. It is deprecated and no longer developed. extraction covers the same job and adds query-relevant highlights, so new integrations should use it.

Moving off livecrawl on POST /v1/search:

extraction is available on POST /v1/search only. GET /v1/search keeps livecrawl for backward compatibility but receives no new features.

Next Steps