What Is Deep Research Evaluation? A Practical Guide to Grading Research Reports

TLDR: Deep research evaluation grades whole research reports, not individual search hits. The dimensions that matter are instruction following, comprehensiveness, completeness, and writing quality, and the reliable way to measure them is pairwise comparison against a reference report. You.com open-sources a harness for exactly this: the ydc-deep-research-evals repository ships a pairwise metric, a business-research dataset, and runnable scripts. This guide covers how the grading works, how to run it, and the failure modes that make naive report scoring misleading.

What is deep research evaluation? It is the practice of scoring the output of a deep research system, a synthesized multi-source report, against a reference report across defined quality dimensions, using a pairwise comparison run by a judge model with order flipping to control position bias.

The distinction from search evaluation matters more than the name suggests. A web search API evaluation scores retrieval: did the right URLs come back for a query. A deep research evaluation scores synthesis: did the system read enough, follow the instructions it was given, and produce a report a professional would ship. A pipeline can ace retrieval and still fail at every one of those, which is why the two evals live in separate harnesses.

Why Is Pairwise Comparison the Default Method?

Pairwise comparison asks a judge model which of two reports is better, then flips the order and asks again, which controls for the single biggest source of noise in judge-based scoring: position bias. The harness in ydc-deep-research-evals runs each trial twice, once in the original order and once with baseline and candidate swapped, specifically to mitigate that bias (repository README, fetched 2026-09-10). The approach was inspired by Google's description of how it evaluated Deep Research on Gemini 2.5 Pro Experimental (Google blog post, linked from the repository).

Absolute scoring, assigning each report a number on a scale, sounds more scientific and behaves worse in practice. Judges drift across runs, apply rubrics inconsistently, and cluster scores toward the middle of whatever scale you hand them. A comparison question is an easier judgment to make reliably, and the flip-and-repeat design turns one noisy judgment into a more stable signal.

Decision framework: use absolute scores when you need to track a single system over time against a fixed rubric, and use pairwise comparison when you need to decide between two systems, which is the actual procurement question. The tradeoff is comparability against drift resistance. If your decision is A or B, run the pairwise harness. If your decision is whether this quarter's build improved over last quarter's, absolute scores against a frozen rubric are cheaper to maintain.

Which Dimensions Does the Harness Grade?

The metric scores four dimensions, each defined precisely in the repository (README, fetched 2026-09-10).

Instruction following measures fidelity to the user's specified instructions and constraints. Did the report respect the requested scope, format, and constraints, or answer a slightly different question. Comprehensiveness measures breadth, whether the range of information addresses the full scope of the request. Completeness measures depth, how thoroughly each covered topic is treated. Writing quality measures clarity, conciseness, logical organization, and readability.

Read the middle two closely, because they pull against each other. A report can be comprehensive and shallow, covering every topic in one paragraph, or deep and narrow, exhausting three topics while ignoring five. Keeping them as separate scores surfaces that tradeoff instead of burying it in one average.

How Do You Set Up the Harness?

The harness is a Python package in the open-source repository with three dependencies: git LFS for the example dataset, the requirements file, and an OpenAI environment for the judge (repository README, fetched 2026-09-10).

git clone https://github.com/youdotcom-oss/ydc-deep-research-evals.git cd ydc-deep-research-evals git lfs install pip install -r requirements.txt export OPENAI_API_KEY=your_openai_api_key export OPENAI_ORGANIZATION_ID=your_openai_org_id

Two things to notice. The judge is an external model, so the evaluation depends on a third-party API key, and judge choice is a real variable you control through the model flag. And the default judge model is o3-mini-2025-01-31, pinned by version string, which means reruns are comparable rather than silently drifting onto whatever the lab's current default is.

How Do You Run an Evaluation?

The script is deep_research_pairwise_evals.py, and the README's invocation shows every flag that matters (fetched 2026-09-10).

python evals/deep_research_pairwise_evals.py \ --input-data datasets/DeepConsult/responses_OpenAI-DeepResearch_vs_ARI_2025-05-15.csv \ --output-dir path/to/output/directory \ --model o3-mini-2025-01-31 \ --num-workers 4 \ --metric-num-workers 3 \ --metric-num-trials 3

The input CSV needs three columns: question, baseline_answer, and candidate_answer. The shipped example dataset, DeepConsult, contains business and consulting prompts covering market analysis, industry evaluation, financial modeling, technology trends, and strategic planning, with baseline and candidate responses collected 2025-05-15. For your own evaluation, replace both columns with your own systems' outputs. The reference does not need to be perfect, it needs to be fixed, because the metric measures the candidate against it, not against an ideal.

Results land as a JSONL file in the output directory, one line per question-answer pair, carrying the original inputs, the per-dimension scores, the aggregate metrics, and the raw evaluation data. Keeping the raw output matters: when a score surprises you, the raw judge transcript is where you find out whether the report failed or the judge misread it.

How Do You Use the Metric Inside Your Own Code?

The metric is importable as a class, so you can wire it into a regression suite instead of running it only as a batch job (README, fetched 2026-09-10).

from evals.metrics.deep_research_pairwise_metric import DeepResearchPairwiseMetric metric = DeepResearchPairwiseMetric( eval_model="o3-mini-2025-01-31", num_trials=3, num_workers=3, ) result = metric.score( question="What are the impacts of climate change on agriculture?", baseline_answer="Your reference answer text...", candidate_answer="Your candidate answer text...", ) print(result.instruction_following.score) print(result.comprehensiveness.score) print(result.completeness.score) print(result.writing_quality.score)

With num_trials set to 3, every judgment runs three times with the order flipped on each pass, so a single lucky or unlucky placement cannot carry the score. That is the difference between a number you can act on and a number you have to re-roll.

What Failure Modes Corrupt a Deep Research Evaluation?

Position bias. Judges favor whichever answer appears first, and the effect is large enough to swamp real quality differences. The harness's flip-and-repeat design exists to neutralize it. If you build your own harness without order flipping, you are measuring placement, not quality.

Dimension collapse. When a judge grades all four dimensions, one loud property, usually length or polish, can bleed across the rubric and drag every dimension up or down together. Detection is cheap: if your four scores move in lockstep across every pair you evaluate, they are measuring one thing four times. Read the raw transcripts and re-check the rubric wording.

Reference drift. Regenerate your baseline mid-evaluation and every candidate score shifts for reasons that have nothing to do with the candidates. Freeze the reference set, version it, and regenerate all candidates against the same frozen baseline when a comparison matters.

Unrepresentative questions. A dataset of consulting prompts measures consulting performance. Before you trust a win, check that the question distribution resembles the queries your system actually receives, or run the eval on a sample of your real traffic.

Where Does the You.com Research API Fit?

The harness grades reports from any system. If one of the systems you are grading is built on the You.com Research API, which returns one-shot cited synthesis across sources, the same pairwise metric applies unchanged: treat its output as the candidate_answer column. The API's research_effort parameter takes lite, standard, deep, exhaustive, or frontier, and the default is standard (you.com/docs/guides/research, accessed September 2026). Effort level is exactly the kind of variable a pairwise harness should measure rather than assume: run the same 20 questions at two tiers, grade them against the same frozen baseline, and you have a dated answer to whether the extra latency buys quality your users notice. Two practical notes for that experiment. The response's content field is Markdown with inline citations like [[1, 2]] that index into a sources array, so strip or keep the markers consistently across both columns, because a judge model will otherwise grade citation formatting instead of research quality. And frontier runs only in background mode and can take minutes, so collect those responses ahead of time rather than inside the eval loop.

For the retrieval layer underneath the report, the web search API evaluation guide covers the search-side benchmark. For grading whole agent workflows rather than reports, the AI agent evaluation guide covers the harness pattern, and the randomness in AI benchmarks guide explains why a single run of any judge-based eval is not a measurement.

Next action: clone the repository, run the shipped DeepConsult dataset through the script unchanged to verify your environment, then build a 20-question CSV from your own domain with your current system in one column and your candidate build in the other. Three trials per pair, frozen reference, raw transcripts saved. That first run on your own questions will teach you more about your research system than any published benchmark.