Hermes Agent + You.com: Web Search Skills That Improve Themselves

TLDR: We gave Hermes Agent web search skills, then measured what its self-improvement loop actually does. Three improvement cycles took our you-research skill from 0.38 to 0.49 mean answer F1 (a 29% relative gain) on a held-out DeepSearchQA sample, with fully-correct answers up from 3 to 5 at a flat tool budget. This is a look at how agent development is becoming a measurable discipline. 

‍We break down the setup— (routing through OpenRouter, scoped GitHub keys, and a You.com MCP server) and share the plugin and eval we're shipping so you can replicate the loop and measure it yourself.

The Challenge: A Bot That Opens Real Integration PRs

We wanted a background agent that could do real developer relations work—discover open source repos where real-time web search would genuinely help, study each codebase, and open a PR that adds the You.com API as an integration. Not a demo, but actual PRs to actual maintainers, following each project's contribution guidelines.

This is a useful stress test for any agent harness because the task has a predictable shape (research, read, plan, code, open the PR) while every repo is different. An agent that can't generalize the pattern burns your budget rediscovering it on every run.

The Setup: Two Models, Scoped Keys, and One MCP Server

We gave Hermes three things.

1. Scoped API keys for GitHub, so it could clone, branch, and open PRs, and nothing else 
2. An OpenRouter key with two open-weight models to choose from (a DeepSeek V4 Flash and a GLM 5.2 model), so inference costs stayed sane 
3. The You.com MCP server, which handed it web search, URL content extraction, and cited research through a single endpoint

We also wrote a skill. Hermes skills are Markdown files that teach the agent when and how to use a capability, and ours was built around the integration PR we already had open to hermes-agent: 

# skills/you-web/SKILL.md
name: you-web
description: Use You.com MCP tools for current web search, URL content extraction, and cited web synthesis.
metadata:
  mcp_servers: '{"you-web":{"url":"https://api.you.com/mcp","auth":"YDC_API_KEY","tools":["you-search","you-contents","you-research"]}}'

The Turn: Hermes Agent Used Our Skill Without Being Told

The first integration PR was slow, and it needed corrections. Here's the honest reason: we were the ones slowing it down. We were updating the skill as we went, telling it which patterns we wanted, and installing the coding hygiene skills we rely on, like TDD. We went back and forth more than we expected and, while it felt like overhead at the time, it was the improvement loop running with humans still in it.

When we asked it to identify the next set of repos and start the work, it finished before we did. The PRs were actually drafted while we were still mid-thought on the follow-up, That's a fun anecdote, but an anecdote is not evidence, so we quantified the same loop with an eval below.

On the next task, Hermes reached for the You.com MCP server on its own. It had folded the capability into the profile.md and memory.md files it maintains about its own environment, effectively storing its self-improvement state where it could easily reference it for future tasks. Every research step ran through the skill even though we never told it to.

Why It Got Better: Skills That Edit Themselves

Hermes reflects on completed tasks and edits its own skills. Give it work with a predictable shape, and every run sharpens the instructions it wrote for itself, within the constraints of its harness and whatever models you handed it. Plenty of harnesses can bolt this on. In Pi, a deliberately bare-bones harness, for example, you can build the same loop as an extension in an afternoon. Hermes ships with it, and that changes how the agent feels by week two.

Two other behaviors earned our trust. 

First was routing. Because we gave it two models, Hermes matched each task to the cheaper one when it could, and costs stayed flat even while it ran autonomously. Other agents we paid for ignored our model selection, defaulted to the expensive option, and kept asking for approval after we had already defined the guardrails. 

Second was scope. Hermes never stepped outside the key scopes we provided.

The Experiment: Distilling the Research API Into a Skill

Watching Hermes improve its own skills pushed us toward a harder question. Could we distill a lightweight version of the You.com Research API, an agentic system that runs multi-step research using our Web Search and Contents APIs as tools, into a plain Markdown skill, running on cheap open-weight models, using nothing but you-search and you-contents? The bar was high: that system ranks first on DeepSearchQA at 83.67% accuracy.

We ran the experiment on Pi across a series of open-weight models. It worked. 

Starting from the shipped you-research skill, three self-improvement cycles—solvers on a small model with only you-search and you-contents, a judge grading against gold answers, and a reflection agent rewriting the skill from its own failures—raised mean answer F1 on a held-out 15-question DeepSearchQA sample from 0.38 to 0.49 (+29% relative), and fully-correct answers from 3 to 5, at a flat tool budget. The improved skill never saw the held-out questions; only the tactics transferred. Review our results.

You can design an agent loop that researches, drafts a skill, evaluates the output, and improves the skill, and you can push a small model surprisingly far when the skill encodes the right search and read patterns. Hermes does a version of this automatically. We had to build it by hand, but building it by hand taught us the real lesson.

Evals Are the Missing Layer

A self-improving skill without an eval is nothing more than a gut feeling. You do eventually get a feel for a model and harness combination just by using it, but a feel can't tell you where the cliff is, and it doesn't survive a budget conversation. A 29% relative gain in three cycles does.

So here's the commitment: every skill and plugin we ship will ship with evals. For Hermes, for Pi, for Claude Code, for OpenCode, for Codex, and for Kimi Code. Eval tooling is how a skill improves in real time inside its own harness—it's how you know when to add deterministic logic at an agent hook instead of another paragraph of prompt and it's how you know when you've hit the ceiling of a model and it's time to post-train an open-weight one, instead of paying frontier prices forever.

This is also what Nous Research got right. They didn't stop at building the improvement loop into Hermes. The same team ships the harness, trains and post-trains open-weight models, and builds the agent so its work can feed back into training. That end-to-end loop—from eval to skill to hook to post-trained model—is the pattern every serious agent team lands on. Agent development is becoming a discipline the way web development once did, and the teams who can measure will come out on top.

What’s New

We're contributing to the Hermes ecosystem.

Our current implementation (pending review from Hermes team) adds a You.com entry to the MCP catalog so search, content extraction, and cited research work out of the box instead of through custom setup.  But you don’t have to wait for this PR to get approved and merged, we have created a plugin that you can install today.

A New Public Plugin

The youdotcom plugin for Hermes bundles five skills and is listed on PyPI as youdotcom-hermes-plugin:

• you-web: current web search, URL content extraction, and cited synthesis
• you-research: multi-hop research and source comparison using you-search plus you-contents
• you-finance: market data, tickers, company financials, and earnings research
• you-discover: finds You.com integration targets, SDK docs, and agent tooling opportunities
• you-free: keyless basic web search, no API key required

The same skills live in the shared repo for Claude Code, Cursor, Codex, GitHub Copilot CLI, Kimi Code, OpenCode, OpenClaw, and Pi, because your search layer shouldn't dictate your harness.

# Install Hermes Agent (Linux, macOS, or WSL2)
curl -fsSL https://raw.githubusercontent.com/NousResearch/hermes-agent/main/scripts/install.sh | bash

# Add the You.com skills
pip install youdotcom-hermes-plugin

# Bring a key for the full toolset, or start keyless with you-free
export YDC_API_KEY=your_key

And, because Hermes improves skills during use, these are starting points. Install you-research, run it against your own workloads, and Hermes will reshape the skill around the way you actually research.

Where You Come In

The day-one review of a self-improving agent is a review of your setup, not the agent. What matters is the slope. Does the tenth run beat the first, is the improvement measurable, and does the agent respect the constraints you set while it climbs? Hermes clears that bar, measurably, which is why we're contributing to it instead of just writing about it.

Install the skills, break them against your own workloads, and tell us where they fall short in the agent-skills repo. If you want You.com merged natively into Hermes, add your voice on the open PRs. And if you want to see how far the underlying APIs go before you wire up an agent at all, start with Building With the Research API.