Retrieve page content

Overview

By default, raw POST /v1/search includes snippets—100–200 words of extracted text per result. The Python SDK 3.6.0 and later sends highlights and knowledge="core" unless you pass knowledge=None and extraction=None. Pass the extraction parameter to attach highlights or full page content. The two modes are mutually exclusive. Pick one per request.

How It Works

Add the extraction object to a POST /v1/search request. The API fetches each result in real time and attaches a contents object to it. The crawl_timeout parameter is a sibling of extraction at the top level of the request body.

When extraction_mode: "highlights", the server rejects crawl_timeout (it only applies to full_page). The Python SDK strips it automatically with a warning before the request goes out, so you can ignore the constraint unless you call the API directly.

Highlights accepts count up to 50. count of 51 or higher returns 422 (highlights supports count up to 50). A request with no extraction accepts count up to 100.

Full-page extraction crawls every web and news result in the response. There is no per-section switch. Control the volume with count. With the default count=10, a call returns up to 10 web + 10 news pages.

Get Highlights

Set extraction_mode to highlights. Each result gains a contents.highlights array of query-relevant excerpts, and snippets is omitted. Do not send extraction_source or crawl_timeout with this mode. Both return 422.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionMode
with You() as you:
  res = you.search(
    query="transformer architecture explained",
    count=5,
    extraction=Extraction(extraction_mode=ExtractionMode.HIGHLIGHTS),
  )
  if res.results and res.results.web:
    for result in res.results.web:
      highlights = result.contents.highlights if result.contents else None
      print(f"{result.title}")
      print(f"  URL: {result.url}")
      if highlights:
        print(f"  Highlights: {len(highlights)} excerpts")
        for h in highlights[:3]:
          print(f"    - {h[:140]}…")
      else:
        print("  (No highlights returned)\n")

Cap Highlight Tokens

By default, highlights for a 10-result search add up to roughly 6,000–7,500 tokens. To keep a response small enough for a latency-sensitive agent, set highlights.total_max_tokens. The budget covers every web result in the response combined, and accepts values from 1024 to 8192.

With total_max_tokens: 1024 and count: 10, a response typically carries five to 10 results with short highlights. With count: 3, it carries three results with longer ones.

curl -X POST https://ydc-index.io/v1/search \
  -H "X-API-Key: $YDC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "AAPL stock price today",
    "count": 3,
    "extraction": {
      "extraction_mode": "highlights",
      "highlights": {
        "total_max_tokens": 1024
      }
    }
  }'

The cap is approximate, so count tokens with your own tokenizer if your prompt has a hard limit. A value outside 1024–8192 returns a 422.

Get Full Page Content

Set extraction_mode to full_page. Each successfully crawled result gains a contents.markdown (or contents.html) field.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode
with You() as you:
  res = you.search(
    query="transformer architecture explained",
    count=5,
    extraction=Extraction(
      extraction_mode=ExtractionMode.FULL_PAGE,
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
    ),
  )
  if res.results and res.results.web:
    for result in res.results.web:
      print(f"{result.title}")
      print(f"  URL: {result.url}")
      if result.contents and result.contents.markdown:
        print(f"  Content ({len(result.contents.markdown)} chars)")
        print(f"  Preview: {result.contents.markdown[:200]}…\n")
      else:
        print("  (No content retrieved)\n")

Choose a Content Source

extraction_source decides whether full-page content comes from the You.com cache, a live crawl, or a mix of the two. Leave it unset to get blend, which is the right default for most pipelines.

cache is the fastest of the three because it never waits on a crawl, and it trades coverage for that speed—expect some results to come back without a contents object. fetch is the slowest and the freshest.

Only pages crawled live are billed, at $1.00 per 1,000 pages. Cached content carries no extraction charge, so cache costs nothing beyond the base call price and blend bills only the pages it crawls live. See Billing for worked examples.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode, ExtractionSource
with You() as you:
  res = you.search(
    query="nvidia earnings",
    count=5,
    extraction=Extraction(
      extraction_mode=ExtractionMode.FULL_PAGE,
      extraction_source=ExtractionSource.FETCH,
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
    ),
  )

Coding Agents That Must Not Live-Crawl

Use extraction_mode: "highlights" when the agent only needs query-relevant excerpts. Do not send extraction_source with highlights. That combination returns 422 (extraction_source is not supported with extraction_mode highlights). For full-page content with no outbound fetch, set extraction_mode to "full_page" and extraction_source to "cache". Results with nothing cached omit contents.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode, ExtractionSource
with You() as you:
  res = you.search(
    query="Python logging.Handler.emit",
    count=5,
    extraction=Extraction(
      extraction_mode=ExtractionMode.FULL_PAGE,
      extraction_source=ExtractionSource.CACHE,
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
    ),
  )

The default blend crawls live on a cache miss, so do not use blend or fetch in locked-down agent environments.

When the agent already has URLs and needs a fetch, call the Contents API instead of extraction_source: "fetch" on the Web Search API.

Full Article Bodies for News

Combine full-page extraction with freshness for breaking news pipelines. Web results are crawled in the same call, so read results.news if articles are all you need.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode, Freshness
with You() as you:
  res = you.search(
    query="semiconductor supply chain",
    freshness=Freshness.WEEK,
    count=5,
    extraction=Extraction(
      extraction_mode=ExtractionMode.FULL_PAGE,
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
    ),
  )
  if res.results and res.results.news:
    for article in res.results.news:
      print(f"{article.title}")
      if article.contents and article.contents.markdown:
        print(article.contents.markdown[:400])
      print()

Control Crawl Timeout

By default the crawler waits up to 10 seconds per page. For latency-sensitive applications, reduce crawl_timeout. For complex or slow-loading pages, increase it (up to 60 seconds). crawl_timeout sits at the top level of the request, not inside extraction, and only applies when extraction_mode: "full_page". Combining it with highlights is invalid.

from youdotcom import You
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode
with You() as you:
  # Low-latency pipeline: only wait 3 seconds per page
  res = you.search(
    query="latest Python releases",
    count=5,
    extraction=Extraction(
      extraction_mode=ExtractionMode.FULL_PAGE,
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
    ),
    crawl_timeout=3,
  )
  if res.results and res.results.web:
    for result in res.results.web:
      status = "crawled" if result.contents and result.contents.markdown else "skipped (timeout)"
      print(f"{result.title} — {status}")

HTML vs Markdown

Pass both formats in extraction_formats to receive contents.markdown and contents.html on every crawled result.

Highlights or Full Page?

A few rules of thumb:

• You want fast, citation-anchored excerpts. Use highlights with a moderate count (10–25). Excerpts land in contents.highlights.
• You need the document body to feed a downstream indexer or synthesizer. Use full_page with extraction_formats: ["markdown"].
• You need rendered HTML for scraping or both formats in the same response. Use full_page with extraction_formats: ["html", "markdown"] to receive both contents.markdown and contents.html per result.
• You already know the URLs. Use the Contents API directly. There is no need to search first.
• You need a live fact, not passages. Add knowledge: "core" for licensed prices, metrics, and forecasts in results.knowledge.

Legacy: livecrawl

Before extraction, page content came from the livecrawl parameter. It still works on both GET and POST /v1/search, so existing integrations keep running. It is deprecated and no longer developed. extraction covers the same job and adds query-relevant highlights, so new integrations should use it.

Moving off livecrawl on POST /v1/search:

extraction is available on POST /v1/search only. GET /v1/search keeps livecrawl for backward compatibility but receives no new features.

Next Steps