Web Search API: Programmatic Access to Real-Time Web Data

TLDR: A web search API gives your application programmatic access to real-time web results as structured JSON, without maintaining a crawler or parsing HTML. This guide covers request and response anatomy, the three content tiers that matter for LLM grounding, domain-level filtering, how to evaluate providers on index freshness and compliance posture, and integration patterns with working code. You.com's Web Search API is one well-documented option. The broader market also includes SerpApi, Brave Search, and DataForSEO, each with different tradeoffs on price, index ownership, and feature depth.

What a Web Search API Actually Is

A web search API is an HTTP endpoint that accepts a query string and returns a ranked list of web documents as machine-readable JSON. Each result carries, at minimum, a URL, a title, a text snippet, and a publication timestamp. The provider operates the crawling and indexing architecture. Your application makes a network request and gets back structured data.

This is categorically different from SERP scraping, where your code fetches the HTML of a search engine results page and parses it. Scraping is fragile (page structure changes break parsers), legally ambiguous under most search engine terms of service, and produces noisy output that requires significant cleanup before an LLM can use it. An API contract gives you a stable schema, metered access, and documented compliance terms.

A scraper that works today may return nothing tomorrow when the provider updates its bot-detection. An API that returns a 200 today will return a 200 tomorrow and the field names will stay where they are.

JSON Response Anatomy

The You.com Web Search API response illustrates the field structure shared by most providers. A POST /v1/search call returns an object with a results key containing two arrays: web and news. The API classifies queries automatically and populates whichever sections apply.

{
  "results": {
    "web": [
      {
        "url": "https://example.com/page",
        "title": "Page Title",
        "description": "A short static description.",
        "snippets": ["Query-relevant excerpt from the page."],
        "page_age": "2025-10-01T00:00:00",
        "favicon_url": "https://ydc-index.io/favicon?domain=example.com"
      }
    ]
  },
  "metadata": {
    "query": "your query here",
    "search_uuid": "a1b2c3d4-...",
    "latency": 0.38
  }
}

Key fields to understand:

• url: The canonical URL of the indexed document, not a redirect through the search engine.
• description: A clean summary of the page that does not change with your query. Useful for display but not reliable for grounding.
• snippets: Query-relevant text extracts. These are what an LLM prompt should use for answering. On You.com, this array can hold multiple passages.
• page_age: An ISO 8601 timestamp for the page's publication date as the index determined it. Use this to filter stale results programmatically.
• metadata.latency: Server-side processing time for the call in seconds. Log it, and you have your own latency distribution instead of a vendor's.

Three Content Tiers

You.com's POST /v1/search endpoint offers three levels of content depth controlled by the extraction object (the older livecrawl parameter is deprecated in its favor). Each level lands in a different response field, which is the detail that trips up parsers.

Mode	What you get	Best for	Response field
Snippets (default, no extraction)	Short, keyword-centered fragments from each result	Quick lookups, ranked lists, entity recognition	snippets
Highlights (extraction_mode: "highlights")	Query-aware passages, only the parts that address the query. snippets is omitted	Agent grounding, RAG pipelines, multi-search tasks	contents.highlights
Full page (extraction_mode: "full_page")	Complete page content as clean Markdown or HTML	Summarization, deep analysis, document-level tasks	contents.markdown or contents.html

For most agent grounding use cases, highlights are the right choice. Full page is token-expensive and it is the only tier that can trigger a live crawl: with the default count=10, a full-page request can crawl up to 20 pages per call (10 web plus 10 news), and each page fetched live is billed on top of the call. extraction.extraction_source controls how many that is: blend (default) serves cached pages free and crawls on a miss, cache never crawls, and fetch crawls everything for maximum freshness. Set extraction.full_page.extraction_formats to ["markdown"] for LLM input. Current per-page rates are on the You.com pricing page, and the math changes quickly at scale. For teams building retrieval-augmented generation systems, the API for RAG overview explains how to layer web search and content extraction into a pipeline.

Basic Integration

Install the Python SDK and run your first query:

pip install youdotcom

from youdotcom import You

with You(timeout_ms=30000) as you:
    results = you.search(query="rust async runtime comparison", count=5)
    for result in results.results.web:
        print(result.title, result.url)
        if result.snippets:
            print(result.snippets[0])

The SDK reads your key from the YDC_API_KEY environment variable, and timeout_ms is worth setting explicitly because the default client timeout is short for requests that crawl pages. The same results are accessible over plain HTTP with the X-API-Key header. New accounts on you.com/platform start with complimentary credits, and the free MCP endpoint at https://api.you.com/mcp?profile=free gives 100 queries per day with no signup for the you-search tool.

cURL and TypeScript Examples

The REST endpoint accepts the same parameters as the SDK. A minimal cURL call with date filtering demonstrates the raw request shape:

curl -X POST https://ydc-index.io/v1/search \
  -H "X-API-Key: $YDC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"rust async runtime","count":5,"freshness":"week"}'

For TypeScript projects, install @youdotcom-oss/sdk from npm and call the same endpoint with typed parameters. Unlike the Python SDK, the TypeScript client does not read the environment for you, so pass the key explicitly, and note that it exposes fields in camelCase (pageAge, not page_age):

import { You } from "@youdotcom-oss/sdk";
import { Freshness } from "@youdotcom-oss/sdk/models";

const you = new You({ apiKeyAuth: process.env.YDC_API_KEY });
const results = await you.search({
  query: "rust async runtime comparison",
  count: 5,
  freshness: Freshness.Week,
});
(results.results?.web ?? []).forEach((r) => console.log(r.title, r.url, r.pageAge));

Domain Filtering and Result Steering

The POST /v1/search endpoint exposes three domain-level controls that are useful for targeted retrieval tasks:

• include_domains: Strict allowlist. Only results from listed domains are returned. Accepts up to 500 domains. Cannot be combined with exclude_domains or boost_domains. Passing either combination returns a 422 error.
• exclude_domains: Block listed domains from results. Useful for filtering out content farms or competitors. Cannot be combined with include_domains.
• boost_domains: Relative ranking boost for listed domains, without filtering others out. Can be combined with exclude_domains.

A concrete use case: a developer documentation agent that should only surface official sources can set include_domains: ["docs.python.org", "developer.mozilla.org", "docs.rs"] and get a scoped index at no extra cost. Passing both include_domains and exclude_domains returns a 422 error, so choose one approach.

The freshness parameter accepts the values day, week, month, year, or a custom date range string formatted as YYYY-MM-DDtoYYYY-MM-DD. Note that when a query's own temporal language is broader than the freshness parameter, the API uses the broader of the two. Setting freshness=day while querying "news this week" will return results scoped to the week.

How Web Search APIs Differ from Each Other

The market breaks down into two structural categories: SERP scrapers and index owners.

SERP scrapers (SerpApi, DataForSEO, Bright Data SERP API) proxy requests through Google, Bing, or other engines and parse the returned HTML. Their output reflects the ranking algorithm of the underlying engine. The advantage is that you get Google's index without owning it. The disadvantage is that your service depends on the upstream engine not detecting and blocking the proxied request, and the response shape can change when the engine changes its HTML structure. Microsoft retired the Bing Search API on August 11, 2025 (per the Microsoft Lifecycle announcement), which forced teams using it to migrate to these proxies or to independent index providers.

Independent index providers (You.com, Brave Search, Exa) crawl and index the web themselves. Response shape is stable under their own control. Ranking criteria and content coverage differ from Google, which can be an advantage for applications that need unbiased results or specialized content coverage, and a disadvantage for tasks where Google's algorithm happens to surface the best content.

A third option is the AI answer API, which wraps web search in a synthesis step: one query returns a natural-language answer with inline citations, with controls for freshness, locale, and domains but no research-effort dial. This is appropriate when the downstream consumer is a human reading a summary rather than an LLM consuming raw retrieval results.

Evaluation Criteria

When selecting a web search API for a production application, evaluate these dimensions:

Index Freshness

How quickly does newly published content appear in results? This matters acutely for news monitoring, financial research, and any agent that needs to know about events from the past hour. Test this empirically: publish a URL you control, wait 30 minutes, and query for it. Providers that claim "real-time" indexing vary widely in what they mean by the term.

Snippet Quality for LLM Consumption

The default snippet is often pulled from the meta description tag or a fixed excerpt, not from the most relevant passage for your query. Evaluate whether the provider offers query-aware highlights before routing to full-page extraction. A highlights mode that returns query-relevant passages costs fewer tokens and adds less latency than crawling every page.

Compliance Posture and Security

Read the terms of service before you ship. Some providers restrict commercial use, resale, or certain use cases explicitly. Verify whether the API agreement covers your product category, geography, and expected volume. SERP proxy services carry additional complexity: the compliance risk of the underlying engine's TOS passes to the API consumer in varying degrees depending on the provider's own agreements.

For enterprise deployments, ask specifically about security attestations, data retention policies, and whether query content is used for model training. You.com documents a Zero Data Retention option that restricts retention of request and response content on the Web Search and Answer APIs. Independent index providers that operate their own architecture generally have clearer data-handling boundaries than proxy services, since no third-party engine touches query content.

Rate Limits and Latency

Rate limits differ significantly across providers and tiers. The You.com free MCP endpoint allows 100 queries per day, and the API returns rate limit headers plus a 429 with Retry-After when you exceed your limit, so build backoff around those rather than a fixed sleep (rate limits documentation). Latency is a function of whether the API is serving indexed results or live-crawling pages. For synchronous agent tasks, measure a provider's median and p95 on your own queries with crawling off, using a per-call timing field like metadata.latency where one exists, before you set a budget. For a deeper look at index freshness and how live crawling affects latency, see the guide to real-time web search API behavior.

Pricing at Scale

Model cost against the shape of your traffic, not the headline per-call number. Three questions decide it: does the provider bill per call or per result (You.com bills per call, with up to 100 results included), is content extraction billed separately per page and does cached content count, and does a synthesis endpoint like a research API bill per task at a rate that varies with effort. You.com publishes all of its rates on one page at you.com/pricing. Read each vendor's page directly, because third-party summaries go stale fast.

Integration Patterns

RAG Retrieval Layer

In a retrieval-augmented generation pipeline, the web search API replaces or supplements a vector database for queries requiring current information. The pattern: receive user query, call search API with highlights mode, inject the top 3-5 snippets into the LLM context window with their source URLs, generate an answer that cites sources. The page_age field allows you to filter results older than a threshold before injection.

AI Agent Tool

Frameworks like LangChain, LlamaIndex, and CrewAI expose tool-calling interfaces where the LLM decides when to invoke search. The You.com MCP server exposes you-search, you-answer, you-contents, and you-research as named tools, among others. Connect any MCP-enabled agent to https://api.you.com/mcp?profile=free for immediate search access without writing integration code. You.com also ships framework-native packages for LangChain, LangGraph, and LlamaIndex, and a crewAI integration through the same MCP server (you.com/docs/integrations, accessed September 2026).

Scheduled Monitoring

For brand monitoring, news alerts, or competitive tracking, the pattern is a scheduled job that queries for specific terms, compares new results against a stored set from the previous run, and triggers a downstream action on new entries. For dedicated news feeds, a news API surfaces headlines and article metadata with a purpose-built endpoint. The freshness=day parameter limits results to the past 24 hours, reducing noise from evergreen content that has not changed.

import os, json
from youdotcom import You

SEEN_FILE = "/var/data/seen_urls.json"

def run_monitor(query: str) -> list[str]:
    seen = set(json.load(open(SEEN_FILE))) if os.path.exists(SEEN_FILE) else set()
    new_urls = []
    with You(timeout_ms=30000) as you:
        results = you.search(query=query, count=10, freshness="day")
        for r in results.results.web:
            if r.url not in seen:
                new_urls.append(r.url)
                seen.add(r.url)
    json.dump(list(seen), open(SEEN_FILE, "w"))
    return new_urls

MCP Access: No-Code Path

For teams that want web search in an agent without writing HTTP client code, the You.com MCP server provides a protocol-native integration. Configure any MCP-enabled IDE (Claude Code, Cursor, VS Code with the Copilot Chat extension, JetBrains) to connect to https://api.you.com/mcp?profile=free and the agent gains a you-search tool at 100 queries per day with no signup. Pass an API key from you.com/platform as an Authorization: Bearer header, or connect with no credentials on a client that supports OAuth 2.1 and sign in through the browser, for the default tool set: you-search, you-contents, you-research, you-answer, you-balance, and you-discover. you-finance is opt-in via https://api.you.com/mcp?tools=you-search,you-finance.

When Not to Use a Web Search API

Web search APIs are not the right tool for every retrieval problem. If your application needs to search a private corpus (internal documentation, proprietary databases), a vector database or keyword search over your own data is more appropriate and avoids sending proprietary query terms to a third-party service. If you need SERP feature parity with Google rankings for SEO analysis tasks (position tracking, SERP feature detection), a SERP scraping API that proxies Google is what you want, not an independent index. And if the question is complex enough to require multi-step reasoning across many sources, You.com's deep research API handles that automatically, running multiple searches and synthesizing a cited answer rather than returning a ranked list you have to process yourself.

Getting Started

The fastest path to a working integration is the You.com quickstart at you.com/docs. Install youdotcom from pip or @youdotcom-oss/sdk from npm, set your YDC_API_KEY, and run the five-line example from the quickstart. The response structure is the same across the SDKs and the REST API apart from the TypeScript client's camelCase field names, so experiments in the REPL translate directly to production code. Full API reference for the search endpoint is at you.com/docs/api-reference/search.