How to Use the You.com Contents API in Python

TLDR: Call the You.com Contents API from Python with one POST to https://ydc-index.io/v1/contents. Send up to 10 URLs and a formats list with your key in the X-API-Key header, then read the markdown from each returned page. A failed crawl returns null content with a 200 status, so check every page, chunk longer lists into batches of 10, and set crawl_timeout and max_age per job.

This guide builds a small Python client for the Contents API with only the standard library: one request function, a null-content check, retries, and a batch runner that handles any number of URLs. The Contents API starts from URLs you already hold. If you start from a query, full page extraction inside the Web Search API attaches contents to search results instead, a path covered in Web Search API in Python. What Is the You.com Contents API? explains when each fits, and What Is a Web Content Extraction API? covers pipeline design and cost. This page is the Python reference for the batch client and its failure handling.

The Contents guide compares the two paths. Contents takes up to 10 URLs per request and costs $1.00 per 1,000 pages. Search with full page extraction returns up to 100 results per search and costs $5.00 per 1,000 searches plus $1.00 per 1,000 pages crawled. Both prices are as of October 2026. Use Contents when you hold the URLs: a watch list, user-submitted links, or citations to check.

How do you call the Contents API from Python?

The Contents API is a POST to https://ydc-index.io/v1/contents, the same host as Web Search, with your key in the X-API-Key header. Create a key on the You.com platform and keep it in YDC_API_KEY. The body takes four fields, summarized from the Contents guide and the API reference.

Field	Meaning	Limits
urls	The pages to fetch.	Up to 10 per request.
formats	markdown (the default), html, or metadata, in any combination.	Request only what you need.
crawl_timeout	How long to wait for each page, in seconds.	1 to 60, default 10.
max_age	The oldest cached copy you accept, in seconds.	0 or greater. Unset accepts any age.

The response is a JSON array with one object per page: url, title, html, markdown, and, when you request metadata, a metadata object holding site_name and favicon_url. The first block sends one batch and raises ContentsError for HTTP failures. Its client timeout is crawl_timeout plus 30 seconds: the docs define crawl_timeout per URL but give no end-to-end response time, so that headroom is a choice. Save the function-defining blocks in this guide, in order, as one file named contents_client.py.

import json
import os
import time
import urllib.error
import urllib.request

BASE_URL = "https://ydc-index.io"  # Contents runs on ydc-index.io


class ContentsError(Exception):
    """An HTTP error response, carrying what a retry decision needs."""

    def __init__(self, status, detail, retry_after=None):
        super().__init__(f"HTTP {status}: {detail}")
        self.status = status
        self.detail = detail
        self.retry_after = retry_after


def fetch_batch(urls, formats=("markdown",), crawl_timeout=10, max_age=None):
    """One request for 1 to 10 URLs. Returns the parsed JSON array, one object per page."""
    if not 1 <= len(urls) <= 10:
        raise ValueError("the Contents API takes 1 to 10 URLs per request")
    body = {"urls": list(urls), "formats": list(formats), "crawl_timeout": crawl_timeout}
    if max_age is not None:
        body["max_age"] = max_age
    request = urllib.request.Request(
        BASE_URL + "/v1/contents",
        data=json.dumps(body).encode("utf-8"),
        headers={"X-API-Key": os.environ["YDC_API_KEY"], "Content-Type": "application/json"},
        method="POST",
    )
    try:
        with urllib.request.urlopen(request, timeout=crawl_timeout + 30) as response:
            return json.load(response)
    except urllib.error.HTTPError as err:
        detail = err.read().decode("utf-8", "replace")[:500]
        raise ContentsError(err.code, detail, err.headers.get("Retry-After")) from None

pages = fetch_batch(["https://you.com/about"], formats=("markdown", "html"))
for page in pages:
    print(page["url"], page.get("title"), len(page.get("markdown") or ""))

What happens when a crawl fails?

A failed crawl is not an HTTP error. If one URL in a batch sits behind a login wall or returns a 404, the API returns null for that page's markdown and html while the rest of the batch succeeds, so the response is still a 200. The docs give a login wall and a 404 as examples, and the page object has no reason field, so you cannot tell a login wall from a slow render by looking at it.

That makes the status code useless for judging pages. Check each page, count the failures per batch, and log the URL, so a blocked source shows up in your metrics instead of as a hole in your index. A null that reaches a chunker fails far from its cause. The helper below treats null, empty, and whitespace-only text alike, because none of them is content.

def has_text(page, fmt="markdown"):
    """True when the requested format came back with text. A failed crawl returns null."""
    return bool((page.get(fmt) or "").strip())

How do you handle errors, rate limits, and retries?

Request failures are separate from empty pages, and the status code tells you which kind. The table follows the error reference.

Status	Meaning	What to do
400, 422	Invalid request parameters or combinations.	Fix the request. Do not retry.
401	Missing, invalid, or expired API key.	Fix the configuration. Do not retry.
402	Out of credits on a keyed account.	Add credits on the platform. Do not retry.
403	The key lacks scope for /v1/contents.	Check the key's scopes. Do not retry.
429	Rate limit exceeded.	Wait for Retry-After when present, otherwise back off, then retry.
500	Internal error, including authentication middleware failures.	Retry a few times, then contact support if it persists.

The default limit for self-serve accounts is 10 requests per second on /v1/contents, and enterprise contracts set their own. Every response carries X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset. The rate limits page recommends reading them to throttle before a 429, and honoring Retry-After when it is present.

def with_retries(operation, attempts=4):
    """Retry 429 and 5xx with backoff. Other 4xx errors are request bugs, so raise at once."""
    for attempt in range(attempts):
        try:
            return operation()
        except ContentsError as err:
            retryable = err.status == 429 or err.status >= 500
            if not retryable or attempt == attempts - 1:
                raise
            try:
                delay = float(err.retry_after)
            except (TypeError, ValueError):
                delay = min(2 ** attempt, 60)
            time.sleep(delay)

The helper retries only 429 and 5xx, honors a numeric Retry-After, and otherwise uses capped exponential backoff. It never retries another 4xx, because repeating a bad request repeats the failure.

How do you extract more than 10 URLs?

Batch to the cap. The guide says a single request with 10 URLs is faster than 10 separate requests, because the API crawls them in parallel. The runner below removes duplicates, splits the list into batches of 10, and sends the batches from a small thread pool.

The docs do not say whether the response array keeps request order, or whether the returned url is exactly the URL you sent, for example after a redirect. The runner therefore keys each batch's results by returned url and sorts every requested URL into one of three outcomes: a page with text, an empty page, or no page under that exact URL. A redirected URL lands in the third group instead of being paired with the wrong content by position.

workers caps concurrency but does not limit the request rate. With four workers and requests that take seconds, you stay far below 10 requests per second, but raising workers on fast pages can exceed it, and the retry helper will then spend its attempts on 429s. A batch that still fails after its retries is recorded whole in failed, because an HTTP 500 gives no way to tell which URL caused it. Rerun those URLs in smaller batches.

from concurrent.futures import ThreadPoolExecutor
from dataclasses import dataclass, field


@dataclass
class Extraction:
    pages: list = field(default_factory=list)      # pages that came back with text
    empty: list = field(default_factory=list)      # requested URLs whose page had no text
    missing: list = field(default_factory=list)  # requested URLs with no page under that exact URL
    failed: list = field(default_factory=list)     # (batch, error) for batches that raised


def extract_all(urls, workers=4, fmt="markdown", **options):
    """Fetch any number of URLs in batches of 10 and sort the outcomes."""
    unique = list(dict.fromkeys(urls))  # drop duplicates, keep order
    batches = [unique[i:i + 10] for i in range(0, len(unique), 10)]
    result = Extraction()

    def run(batch):
        try:
            return batch, with_retries(lambda: fetch_batch(batch, formats=(fmt,), **options)), None
        except ContentsError as err:
            return batch, [], err

    with ThreadPoolExecutor(max_workers=workers) as pool:
        for batch, returned, err in pool.map(run, batches):
            if err is not None:
                result.failed.append((batch, str(err)))
                continue
            by_url = {page.get("url"): page for page in returned}
            for url in batch:
                page = by_url.get(url)
                if page is None:
                    result.missing.append(url)
                elif has_text(page, fmt):
                    result.pages.append(page)
                else:
                    result.empty.append(url)
    return result

How do you tune crawl_timeout and max_age?

crawl_timeout is per URL, and the guide frames it as speed against completeness: 5 to 10 seconds for fast pages, 20 to 30 for JavaScript-heavy ones. The docs do not list a short timeout as a cause of null content, but it is the cheapest setting to try when a page you know exists comes back empty. The entry point below therefore runs two passes: everything at the default of 10 seconds, then only the empty pages at 30. The docs do not say whether failed crawls are billed, so budget the second pass as extra pages.

max_age is the oldest cached copy you will accept. By default the API may return cached content of any age, which is fast and wrong for a job whose purpose is detecting change, because a stale cache hit looks exactly like an unchanged page. Set it from each job's real freshness requirement, not once per pipeline.

Setting	Use it when
crawl_timeout 5 to 10	The pages are fast and mostly static.
crawl_timeout 20 to 30	The pages render with JavaScript, or the first pass returned them empty.
max_age=86400	A daily job needs content no older than 24 hours.
max_age=0	You always want a fresh fetch and accept the extra latency.
max_age unset	Freshness does not matter and you want lower latency.

Cost follows pages. At $1.00 per 1,000 pages as of October 2026, per the billing page, 5,000 URLs cost $5.00, and a daily check of 200 pages costs $6.00 over 30 days. Request only the formats you need, since the guide notes that each format adds processing time. The entry point reads URLs from a file and prints one JSON object per page.

import sys


def main(argv):
    with open(argv[0], encoding="utf-8") as handle:
        urls = [line.strip() for line in handle if line.strip()]
    first = extract_all(urls, crawl_timeout=10)
    retry = extract_all(first.empty, crawl_timeout=30)  # slow pages get one more chance
    pages = first.pages + retry.pages
    for page in pages:
        print(json.dumps({"url": page["url"], "title": page.get("title"), "markdown": page["markdown"]}))
    print(
        f"{len(pages)} pages, {len(retry.empty)} still empty, "
        f"{len(first.missing) + len(retry.missing)} missing, "
        f"{len(first.failed) + len(retry.failed)} failed batches",
        file=sys.stderr,
    )


if __name__ == "__main__":
    main(sys.argv[1:])

Should you use the Python SDK instead?

The official SDK, youdotcom, wraps the endpoint as you.contents() and returns typed page objects. Version 3.5.0, released September 22, 2026 and published on PyPI, requires Python 3.10 or later and reads YDC_API_KEY from the environment, as the Python SDK page documents. Pin the version. Each call sends one POST, so batching, concurrency, and per-URL accounting stay your job. Choose the SDK for typed fields that track the API, and the standard-library client when you want that control in one dependency-free file.

# pip install youdotcom==3.5.0   (Python 3.10 or later)
from youdotcom import You
from youdotcom.models import ContentsFormats

with You(timeout_ms=60_000) as you:  # reads YDC_API_KEY from the environment
    pages = you.contents(
        urls=["https://you.com/about", "https://www.python.org"],
        formats=[ContentsFormats.MARKDOWN],
        crawl_timeout=15,
        max_age=86_400,
    )
    for page in pages:
        text = page.markdown or ""
        print(page.url, page.title, len(text))

Set timeout_ms. Without it, the client falls back to the HTTPX default of five seconds, shorter than a crawl_timeout of 10 or more. In an offline run against a mock that answered after six seconds, a call without timeout_ms failed with a ReadTimeout after 5.0 seconds. Failed pages come back with markdown set to None, so keep the or "" guard. The metadata format is deprecated in 3.5.0 and emits a DeprecationWarning, so skip it unless you need the site name and favicon.

What should you test before shipping?

Test the client without paying for pages. The code blocks were syntax-checked, and the client blocks, saved in article order as one file, passed 20 unit tests on Python 3.9 against a local mock server shaped like the documented response: null content, a slow page that needs a longer timeout, a redirected URL, a reversed response order, duplicate URLs, a batch that always fails, and 401, 403, 422, 429, and 500 errors. Eight deliberate breakages of the code were each caught: matching by position, batches of 11, no deduplication, treating null as text, retrying every 4xx, swallowing a failed batch, dropping max_age=0, and a client timeout equal to crawl_timeout. The SDK snippet ran against the same mock, also on Python 3.9, below the SDK's stated minimum, so read that as a check of the call shape. No real key and no live call were used.

That verifies client logic, not live crawl success, latency, billing, or rate-limit behavior. Run an authorized sample against each domain you plan to ingest, read the Markdown by hand, then work through this list:

• Fixtures: keep one response each for null markdown, shuffled order, a redirected URL, and duplicate URLs.
• Timeouts: keep the client timeout above crawl_timeout, and give empty pages a second pass.
• Freshness: choose max_age per job, and store it with the page.
• Failures: alert on the empty, missing, and failed-batch counts of every run.
• Cost: multiply pages, retries included, by the per-page price, and recheck the billing page, because prices change.

Related Guides

• What Is the You.com Contents API? Clean Page Content From Any URL
• What Is a Web Content Extraction API? How to Build a Pipeline With the You.com Contents API
• Web Search API in Python: A Practical Guide With the You.com SDK
• What Is a Price Monitoring API? How to Build One With the You.com Contents API