Search for AI Agents: Paid APIs vs a Self-Hosted Instance

An agent that cannot search is limited to what its model already knows, a snapshot of the web from some months ago. Give it a search tool and it can check a price, read a changelog, find the current version of a library, or confirm that a page still says what the training data said it did. At that point search stops being a feature and becomes infrastructure, with the questions any infrastructure gets: cost, reliability, and what it exposes. The choice between a paid search API and a self-hosted metasearch instance is arithmetic and privacy, not ideology. We run both, and this is how we decide.

What does an agent actually do with search?

A search tool is a function the model can call. It writes a query, gets back titles, URLs and snippets, and either answers from those snippets or fetches two or three pages for the full text. One research task in our automations is usually between five and forty queries: the first search, a few follow-ups on names it found, and a retry or two when a snippet was not enough. That is the budget line. Output tokens are the visible cost of an agent; search calls are the quiet one.

What does a paid search API cost per query?

Four providers we priced today, all keyed APIs that return JSON:

• Brave Search API: $5 per 1,000 requests, 50 queries per second, with $5 of credit every month.
• Serper: $1 per 1,000 queries in the smallest prepaid pack, $0.30 in the largest, and 2,500 free queries once.
• Tavily: one credit per basic search at $0.008 per credit, so $8 per 1,000, with 1,000 free credits a month.
• Exa: $7 per 1,000 searches on the auto and fast modes, $4 on lite, with $10 of credit a month.

The headline rate hides three things. Serper's credits expire after six months, so its $0.30 rate means $3,750 up front and 12.5 million queries consumed in half a year. Tavily charges two credits for an advanced search, which doubles its $8. Exa bills page contents separately, at $1 per 1,000 pages for each content type.

One option that used to be the default is gone for new projects. Google's Programmable Search JSON API is closed to new customers, and existing ones have until 1 January 2027 to move. It cost $5 per 1,000 queries, with 100 free a day.

What does a self-hosted instance cost?

A SearXNG instance holds no index of its own. It forwards a query to several upstream engines in parallel, parses the HTML that comes back, merges and de-duplicates the results, and returns them as a page or as JSON. The work is network-bound and bursty, so the hardware bar is low. Hetzner's CX23 costs €5.99 a month excluding VAT, with 2 vCPU, 4 GB of RAM, 40 GB of NVMe and 20 TB of traffic, though its page marked that plan unavailable when we checked; the smallest DigitalOcean droplet is $6 for 1 GB of memory, 25 GB of disk and 1 TB of transfer. Below we treat the server as $7 a month and ignore the domain, which you probably already own.

What the price does not include is attention: engines change their markup, add challenges and block addresses, and each of those is an evening.

Where is the break-even?

Stated assumptions: one search is one billed query, with no retries; the instance costs $7 a month at any volume; all figures are list prices from the pages linked above, before tax.

• 1,000 queries a month: Brave $5, Serper $1, Tavily $8, self-hosted $7.
• 10,000 queries a month: Brave $50, Serper $10, Tavily $80, self-hosted $7.
• 30,000 queries a month: Brave $150, Serper $30, Tavily $240, self-hosted $7.
• 100,000 queries a month: Brave $500, Serper $100, Tavily $800, self-hosted $7.

The first line decides most projects. At a thousand queries a month the free tiers cover you almost everywhere, and the instance is the most expensive option on the list. The server only wins once the bill passes $7, which against Brave is about 1,400 queries a month, roughly 45 a day. Against Tavily it is about 900. Against Serper's dollar it is 7,000, and against its best prepaid rate, 23,000.

So the money break-even is low, and at 30,000 queries a month the gap is not subtle: $150 against $7. That figure assumes the instance answers. It leaves out retries when an engine is suspended, the evening spent fixing a parser, and the second instance if search sits on a critical path. We move when the volume is steady, the queries are internal, and the box already exists for something else.

What does a paid search API learn about your queries?

Every request carries the query text, a timestamp, your API key and the address of the machine that sent it. That is enough to reconstruct what your agents worked on, in order and at what time of day. The practical rule for client work is shorter than the legal one: assume the vendor can read every query your agent sends. For public research that is usually fine; for anything under an NDA it usually is not.

Tavily's privacy policy says it collects "your query data and any documents you upload", may share query data with third-party index providers when its own index cannot retrieve the content, and may use "certain portions of your query data to improve our responses to future queries" unless your contract says otherwise. Brave's API is tied to an account key, and we could not confirm a retention statement on its pricing page, so read its terms before assuming queries are discarded.

What does a self-hosted instance still leak?

Less, but not nothing. SearXNG's documentation says it sends no cookies to upstream engines and generates a random browser profile for every request, then states the limit plainly: "the IP address used will be the IP of the instance". The query still leaves your network and still reaches Google, Bing, Brave and DuckDuckGo, from one address that every search you make shares.

That concentration cuts both ways. Engines cannot tie a query to a person, but they can see one source issuing many queries in a pattern that does not look like a browser, which is what their bot detection notices. The same page warns that a public instance can be abused until engines impose CAPTCHAs or ban its address. Self-hosted is not anonymous. It is unlogged by a third party, and only as long as you keep it that way: your host still sees the traffic, and the instance logs whatever you configure. If queries must be unattributable rather than merely unprofiled, the documented route is a proxy or Tor, which adds latency and makes blocking more likely.

Which engines can an instance aggregate?

SearXNG supports up to 261 search services and enables 81 by default, across web, images, news, video, maps, code, science and social categories. The setup we publish enables seven: Google, DuckDuckGo, Brave, Bing, Wikipedia, GitHub and YouTube. That is a choice, not a ceiling. Every extra engine adds a request to each query, another parser that can break, and another chance of a challenge.

The code engines surprise people. Stack Overflow, GitHub, MDN and the package registries are separate services, so an agent asked about a Python traceback can query the places where that traceback is discussed instead of hoping a general index surfaced the right thread. One operational detail catches everyone once: JSON output is off by default, and a request for a format that is not enabled returns 403.

What breaks first: rate limits and blocking

Upstream engines are not a free API, and they treat automated traffic accordingly. SearXNG ships with suspension times for each refusal: an HTTP 429 suspends that engine for an hour, a CAPTCHA for a day, a Google CAPTCHA for a week, a Cloudflare CAPTCHA for fifteen days. While an engine is suspended the instance keeps answering, with fewer results and no error. An agent that never checks where its results came from will not notice that half its sources have disappeared.

The instance needs protection in the other direction too. SearXNG's limiter exists because bot traffic through your instance makes it look like a bot to the engines, and it needs a Valkey database to track addresses and apply limits. Without it, an open endpoint is a way for someone else to get your server blocked.

Does it cache, and is the quality the same?

SearXNG has no built-in cache for results; its search settings cover failures and suspensions, not storage. Caching is something you add, and our setup uses a local Redis. Worth doing for an agent that asks the same question twice, and worth being careful with for anything time-sensitive: a cached answer to "latest release" is a wrong answer with a timestamp on it.

Quality is the honest limit. A metasearch instance merges lists that each engine ranked for its own product, with no shared ranking model, no click data and no freshness guarantee. For navigational queries and breaking news it comes close to a commercial API. For long-tail technical questions, exact phrases, and anything where the tenth result matters, it is worse, occasionally visibly so. Self-hosted search is a second source and a fallback, not a drop-in replacement for a tuned commercial index.

How we choose between them

• Under 1,000 queries a month: use a free tier and spend the effort on the prompt.
• Steady internal volume, data you would rather not hand to a vendor: self-host, and keep one paid key for the days an engine is suspended.
• A product feature where a missing result costs money: a paid API with support, cached, and the instance behind it for when the quota runs out.
• Regulated data: self-host, and still assume the upstream engines see the query text.

What we run is the hybrid: one paid key for anything user-facing, one instance for internal research, routing by query type.

Can it be set up in one command?

We published the setup we use, SearXNG plus the OpenClaw search plugin, at searxng.danubelabs.net. It installs the instance with the seven engines above, a local Redis cache and a systemd service, then registers the search plugin in OpenClaw's configuration after backing it up. The page estimates about twenty minutes on Ubuntu 20.04 or Debian 11 and later, the project is MIT licensed, and it says outright that requests still reach the upstream engines from your server's address.

Danube Labs is a founder-led software and AI studio in Bucharest. We build agents that need to look things up, and we care which infrastructure they look them up through. If you are wiring search into a product and want a second opinion on the cost or the data path, see our AI systems work or book a free intro call.