> For clean Markdown of any page, append `.md` to the page URL. > Documentation index: https://you.com/docs/llms.txt (section indexes: append `/llms.txt` to any section URL). > Search these docs: Docs MCP at https://you.com/docs/_mcp/server (`searchDocs`, no API key). > Call live You.com APIs: Product MCP at https://api.you.com/mcp (keyless `?profile=free` exposes `you-search` and `you-discover`. Research and finance use `/mcp/research`, `/mcp/finance`, or `?tools=` on `/mcp`). To pick an API or integration path, call `you-discover` on that server instead of guessing. > OpenAPI: https://you.com/docs/openapi.json—auth header `X-API-Key`, env `YDC_API_KEY`. # Contents API > Web Search, Answer, Contents, Research, and Finance Research APIs that give your AI app real-time web intelligence. Start with free credits. ## Docs - [Contents API Quickstart](https://you.com/docs/guides/contents/quickstart.md): Pass a list of URLs to the Contents API and get back clean Markdown or HTML, ready for LLM consumption. > **Note:** This page contains both a page directory (above) and the landing page content (below). The page directory is generated for agent use and does not appear on the landing page. > For clean Markdown of any page, append `.md` to the page URL. > Documentation index: https://you.com/docs/llms.txt (section indexes: append `/llms.txt` to any section URL). > Search these docs: Docs MCP at https://you.com/docs/_mcp/server (`searchDocs`, no API key). > Call live You.com APIs: Product MCP at https://api.you.com/mcp (keyless `?profile=free` exposes `you-search` and `you-discover`. Research and finance use `/mcp/research`, `/mcp/finance`, or `?tools=` on `/mcp`). To pick an API or integration path, call `you-discover` on that server instead of guessing. > OpenAPI: https://you.com/docs/openapi.json—auth header `X-API-Key`, env `YDC_API_KEY`. # Contents API Overview > **Tip** > > **Add You.com to your agent or IDE via MCP**—connect at `https://api.you.com/mcp`. Try search with no signup at `https://api.you.com/mcp?profile=free` (`you-search` and `you-discover`). Research and finance use `/mcp/research` and `/mcp/finance`, or `?tools=` on `/mcp`. [MCP Server guide →](/docs/agents/mcp-server) > **Install the Docs MCP server** > > The Docs MCP server lets your agent search over the entirety of these docs. Free to use. > > Install `https://you.com/docs/_mcp/server`. See the [Docs MCP Server guide](/docs/agents/docs-mcp-server) to learn more. --- ## What Is the Contents API? The Contents API extracts clean HTML or Markdown content from a given URL. Pass it a list of URLs and get back the full page content for each, ready for LLM consumption—no parsing, no HTML noise, no browser automation required. --- ## Endpoints Each API has its own endpoint. Every request sends your API key in the `X-API-Key` header. | API | Endpoint | Guide | API reference | | -------------------- | -------------------------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------- | | Web Search API | `POST https://ydc-index.io/v1/search` (also `GET`) | [Overview](/docs/guides/search) | [/v1/search](/docs/api-reference/search/v1-search) | | Contents API | `POST https://ydc-index.io/v1/contents` | [Overview](/docs/guides/contents) | [/v1/contents](/docs/api-reference/contents) | | Answer API | `POST https://api.you.com/v1/answer` | [Overview](/docs/guides/answer) | [/v1/answer](/docs/api-reference/answer/v1-answer) | | Research API | `POST https://api.you.com/v1/research` | [Overview](/docs/guides/research) | [/v1/research](/docs/api-reference/research/v1-research) | | Finance Research API | `POST https://api.you.com/v1/finance_research` | [Overview](/docs/guides/finance-research) | [/v1/finance_research](/docs/api-reference/finance-research/v1-finance_research) | The Answer, Research, and Finance Research APIs run only on `api.you.com`. Sending one of those paths to `ydc-index.io` returns `403` with `{"message": "Missing Authentication Token"}`, even when the key is valid. --- ## How It's Different From Full Page Extraction The Contents API and full page extraction in the Web Search API both extract full page content, but they serve different workflows: | | Contents API | Web Search API + full page extraction | | ------------------ | --------------------------- | -------------------------------------------------------- | | **Starting point** | You already know the URLs | You have a query, not URLs | | **Use case** | Fetch known pages on demand | Enrich search results with full content | | **URL source** | You provide them | You.com search discovers them | | **Batch size** | 10 URLs per request | Up to 100 results per search | | **Pricing** | \$1.00 per 1000 pages | \$5.00 per 1000 searches + \$1.00 per 1000 pages crawled | Use the Contents API when you have a list of specific URLs you want to read. Use `extraction_mode: "full_page"` when you want full content returned alongside search results in one go. See [Retrieve page content](/docs/guides/search/retrieve-page-content). --- ## What You Get Each URL in your request returns a structured object: ```json [ { "url": "https://competitor.com/pricing", "title": "Pricing — Competitor Inc.", "markdown": "# Pricing\n\n## Starter Plan\n$49/month...", "html": "...", "metadata": { "site_name": "Competitor Inc.", "favicon_url": "https://ydc-index.io/favicon?domain=competitor.com&size=128" } } ] ``` You control which formats are returned via the `formats` parameter—request `markdown`, `html`, and/or `metadata` in any combination. --- ## Key Features ### Any URL, on demand Pass up to 10 URLs in a single request. The API crawls them all in parallel and returns the content. No need to manage a headless browser or deal with raw HTML yourself. ### LLM-ready Markdown The `markdown` format strips navigation menus, ads, footers, and other boilerplate. You get actual content of the page—ready to drop into a prompt. ### Configurable timeout Use `crawl_timeout` (1–60 seconds) to balance speed vs. completeness. For fast pages: 5–10 seconds. For heavy JavaScript-rendered pages: 20–30 seconds. ### Cache Freshness Control By default, the Contents API may return cached page content to improve latency. Use `max_age` to control how fresh that cache must be. Pass a value in seconds and any cached content older than the threshold is ignored, forcing a fresh fetch of the page. For example, `max_age=86400` ensures content is no older than 24 hours, while `max_age=0` always bypasses the cache and re-fetches the page. Leave `max_age` unset (the default) to accept cached content regardless of age. ### Metadata extraction Request `metadata` alongside content to get the page's site name and favicon URL—useful for building UIs that display source attribution. --- ## Parameters | Parameter | Type | Required | Description | | --------------- | ---------------- | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `urls` | array of strings | Yes | The URLs to fetch content from | | `formats` | array of strings | No | Content formats to return: `markdown`, `html`, `metadata` (default: `markdown`) | | `crawl_timeout` | number | No | Per-URL timeout in seconds, between 1 and 60 (default: 10) | | `max_age` | integer | No | Maximum allowed age of cached content in seconds. Must be 0 or greater. When set, cached content older than this threshold is ignored and the page is re-fetched. Default: `null` (no age limit) | [View full API reference](/docs/api-reference/contents) --- ## Common Use Cases ### Competitive intelligence Monitor competitor pricing, feature, or blog pages. Fetch the content on a schedule, feed it to an LLM, and surface meaningful changes—without manual checking. ```python from youdotcom import You from youdotcom.models import ContentsFormats competitor_pages = [ "https://competitor-a.com/pricing", "https://competitor-b.com/pricing", "https://competitor-c.com/features", ] with You() as you: pages = you.contents( urls=competitor_pages, formats=[ContentsFormats.MARKDOWN], crawl_timeout=15, ) for page in pages: print(f"\n--- {page.title} ---") # Feed page.markdown into your LLM for summarization or diff print(page.markdown[:500]) ``` ### Knowledge base ingestion You have a list of authoritative sources—documentation pages, whitepapers, internal wikis. Fetch them all, convert to clean Markdown, and index into your vector store. ```python from youdotcom import You from youdotcom.models import ContentsFormats # Known authoritative sources to index source_urls = [ "https://docs.example.com/api-reference", "https://docs.example.com/authentication", "https://docs.example.com/rate-limits", ] with You() as you: pages = you.contents( urls=source_urls, formats=[ContentsFormats.MARKDOWN, ContentsFormats.METADATA], ) documents = [] for page in pages: documents.append({ "source": page.url, "title": page.title, "content": page.markdown, }) # Index document into your vector store here ``` ### Research assistant Give users the ability to ask questions about specific URLs. Fetch the page content on the fly and feed it as context into your LLM—turning any URL into a searchable document. ```python from youdotcom import You from youdotcom.models import ContentsFormats def fetch_url_context(url: str) -> str: with You() as you: pages = you.contents(urls=[url], formats=[ContentsFormats.MARKDOWN]) return pages[0].markdown if pages else "" # User asks: "Summarize this page for me" url = "https://example.com/long-report" context = fetch_url_context(url) prompt = f"Summarize the following page content:\n\n{context}" # Pass prompt to your LLM ``` --- ## Best Practices ### Request only the formats you need Each format adds processing time. If you only need Markdown for LLM consumption, don't request `html`. If you don't need site metadata for your UI, skip `metadata`. ### Batch your URLs A single request with 10 URLs is faster than 10 separate requests. The API processes them in parallel. ### Set `crawl_timeout` based on the target site For simple static pages, 5–10 seconds is usually enough. For JavaScript-heavy pages (SPAs, dashboards), increase to 20–30 seconds to give the renderer time to complete. ### Handle partial failures gracefully If one URL in a batch fails to crawl (e.g., it's behind a login wall or returns a 404), the API returns `null` for its `markdown` and `html` fields. Always check before processing: ```python for page in pages: if page.markdown: # Process content pass else: print(f"Failed to fetch: {page.url}") ``` --- ## Pricing **\$1.00 per 1,000 pages** See the [Billing page](/docs/administration/billing#contents-api) for complete pricing details. **What's included:** * Batch multiple URLs per request * Clean Markdown or raw HTML output * Python SDK, MCP Server, and REST API access * No browser automation or HTML parsing required For volume discounts, annual pricing, or enterprise features, visit [you.com/pricing](https://you.com/pricing) or contact [api@you.com](mailto:api@you.com). --- ## Next Steps #### [Quickstart](/docs/guides/contents/quickstart) Make your first Contents API request #### [API Reference](/docs/api-reference/contents) Full parameter reference, request/response schemas, and error codes #### [Web Search API Overview](/docs/guides/search) Pair search results with full page extraction to get full content alongside real-time web data #### [Python SDK](/docs/integrations/python-sdk) Use the official SDK for cleaner integration