Retrieve page content

Overview

By default, search results include snippets—100–200 words of extracted text per result. Pass the extraction parameter to ask for richer content: full-page Markdown and HTML, or query-relevant highlights.

This unlocks:

• Deep RAG with full document context
• Knowledge base construction from live web data
• Comprehensive content synthesis across sources
• Full article bodies for news results

Two modes are available. Pick the one that matches the latency and token budget of your pipeline.

If you want query-ranked excerpts rather than whole pages, use extraction_mode: "highlights". The two modes are mutually exclusive—pick one per request.

How It Works

Add the extraction object to a POST /v1/search request. The API fetches each result in real time and attaches a contents object to it. The crawl_timeout parameter is a sibling of extraction at the top level of the request body.

When extraction_mode: "highlights", the server rejects crawl_timeout (it only applies to full_page). The Python SDK strips it automatically with a warning before the request goes out, so you can ignore the constraint unless you call the API directly.

Full-page extraction crawls every web and news result in the response. There is no per-section switch. Control the volume with count. With the default count=10, a call returns up to 10 web + 10 news pages.

Get Highlights

Set extraction_mode to highlights. Each result gains a contents.highlights array of query-relevant excerpts. Snippets are omitted in this mode—the search returns the ranked highlights instead.

1
from youdotcom import You
2
from youdotcom.models import Extraction, ExtractionMode
3
4
with You() as you:
5
  res = you.search(
6
    query="transformer architecture explained",
7
    count=5,
8
    extraction=Extraction(extraction_mode=ExtractionMode.HIGHLIGHTS),
9
  )
10
11
  if res.results and res.results.web:
12
    for result in res.results.web:
13
      highlights = result.contents.highlights if result.contents else None
14
      print(f"{result.title}")
15
      print(f"  URL: {result.url}")
16
      if highlights:
17
        print(f"  Highlights: {len(highlights)} excerpts")
18
        for h in highlights[:3]:
19
          print(f"    - {h[:140]}…")
20
      else:
21
        print("  (No highlights returned)\n")

Get Full Page Content

Set extraction_mode to full_page. Each successfully crawled result gains a contents.markdown (or contents.html) field.

1
from youdotcom import You
2
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode
3
4
with You() as you:
5
  res = you.search(
6
    query="transformer architecture explained",
7
    count=5,
8
    extraction=Extraction(
9
      extraction_mode=ExtractionMode.FULL_PAGE,
10
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
11
    ),
12
  )
13
14
  if res.results and res.results.web:
15
    for result in res.results.web:
16
      print(f"{result.title}")
17
      print(f"  URL: {result.url}")
18
      if result.contents and result.contents.markdown:
19
        print(f"  Content ({len(result.contents.markdown)} chars)")
20
        print(f"  Preview: {result.contents.markdown[:200]}…\n")
21
      else:
22
        print("  (No content retrieved)\n")

Full Article Bodies for News

Combine full-page extraction with freshness for breaking news pipelines. Web results are crawled in the same call, so read results.news if articles are all you need.

1
from youdotcom import You
2
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode, Freshness
3
4
with You() as you:
5
  res = you.search(
6
    query="semiconductor supply chain",
7
    freshness=Freshness.WEEK,
8
    count=5,
9
    extraction=Extraction(
10
      extraction_mode=ExtractionMode.FULL_PAGE,
11
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
12
    ),
13
  )
14
15
  if res.results and res.results.news:
16
    for article in res.results.news:
17
      print(f"{article.title}")
18
      if article.contents and article.contents.markdown:
19
        print(article.contents.markdown[:400])
20
      print()

Get Both Markdown and HTML

Pass both formats in extraction_formats to receive contents.markdown and contents.html on every crawled result.

$
curl -X POST https://ydc-index.io/v1/search \
>
  -H "X-API-Key: $YDC_API_KEY" \
>
  -H "Content-Type: application/json" \
>
  -d '{
>
    "query": "quantum computing breakthroughs",
>
    "count": 5,
>
    "extraction": {
>
      "extraction_mode": "full_page",
>
      "full_page": {
>
        "extraction_formats": ["markdown", "html"]
>
      }
>
    }
>
  }'

Control Crawl Timeout

By default the crawler waits up to 10 seconds per page. For latency-sensitive applications, reduce crawl_timeout. For complex or slow-loading pages, increase it (up to 60 seconds). crawl_timeout sits at the top level of the request, not inside extraction, and only applies when extraction_mode: "full_page". Combining it with highlights is invalid.

1
from youdotcom import You
2
from youdotcom.models import Extraction, ExtractionFormat, ExtractionMode
3
4
with You() as you:
5
  # Low-latency pipeline: only wait 3 seconds per page
6
  res = you.search(
7
    query="latest Python releases",
8
    count=5,
9
    extraction=Extraction(
10
      extraction_mode=ExtractionMode.FULL_PAGE,
11
      full_page={"extraction_formats": [ExtractionFormat.MARKDOWN]},
12
    ),
13
    crawl_timeout=3,
14
  )
15
16
  if res.results and res.results.web:
17
    for result in res.results.web:
18
      status = "crawled" if result.contents and result.contents.markdown else "skipped (timeout)"
19
      print(f"{result.title} — {status}")

HTML vs Markdown

Highlights or Full Page?

A few rules of thumb:

• You want fast, citation-anchored snippets. Use highlights with a moderate count (10–25). Excerpts land in contents.highlights.
• You need the document body to feed a downstream indexer or synthesizer. Use full_page with extraction_formats: ["markdown"].
• You need rendered HTML for scraping or both formats in the same response. Use full_page with extraction_formats: ["html", "markdown"] to receive both contents.markdown and contents.html per result.
• You already know the URLs. Use the Contents API directly. There is no need to search first.

Already Have URLs?

If you have a list of URLs and don’t need to search first, use the Contents API directly. It accepts URLs without a query and returns the same markdown or html content.

Legacy: livecrawl

Before extraction, page content came from the livecrawl parameter. It still works on both GET and POST /v1/search, so existing integrations keep running. It is deprecated and no longer developed. extraction covers the same job and adds query-relevant highlights, so new integrations should use it.

Moving off livecrawl on POST /v1/search:

extraction is available on POST /v1/search only. GET /v1/search keeps livecrawl for backward compatibility but receives no new features.

Next Steps