agent-fetch — Reliable web access for AI agents

The problem

Half the web fights back

Your agents need to read web pages, but programmatic fetching has gotten adversarial. You can throw a headless browser at every request or route through a scraping API, but that's slow, expensive, and uses datacenter IPs that sites are trained to block.

Bot detection

Cloudflare challenges, CAPTCHAs, and fingerprint checks that reject anything that isn't a real browser

Empty responses

SPAs and SSR pages that return empty <div>s until JavaScript executes client-side

Paywalls

Content you already pay for, locked behind login gates your agents can't pass

Datacenter IPs

Scraping APIs route through flagged IP ranges. Your home IP is a residential visitor.

How it works

Cheapest method first, escalate only when needed

Each response passes acceptance checks automatically. If the result is blocked, empty, or paywalled, the next strategy fires. No wasted browser launches for pages that work with a plain fetch.

01 fetch Plain HTTP request. Fastest, cheapest. Works for most static pages and APIs.

02 jsdom Runs JavaScript locally to handle client-rendered pages without a browser.

03 plugins Third-party rendering (e.g. scrape.do) for pages with aggressive bot detection.

04 agent-browser Headless Chrome with your cookies and sessions. The heavy option, used last.

Each response validated: word count, blocked-page patterns, paywall detection. Bad results caught before they reach your agent.

Your machine, your access

Sites see a residential visitor, not a datacenter

agent-fetch runs locally. Your requests come from your IP address, with your browser sessions. This solves a category of problems that cloud scraping APIs can't.

>

Your IP address

Residential IPs aren't on blocklists. Sites that block AWS and GCP ranges let your home connection through.

>

Your subscriptions

Set up a browser profile, log into Stratechery or your company wiki once, and agent-fetch reuses those sessions for every request.

>

Your network

Internal tools, staging environments, localhost services. If you can reach it, your agents can too.

Agent integration

POST a URL, get markdown back

Start the HTTP server and any agent on your network can fetch web content. Put it behind Tailscale so it's reachable from anywhere on your tailnet without exposing it to the public internet.

Agent

your tool

→

Server

agent-fetch

→

Output

markdown

Start the server

agent-fetch server --host 0.0.0.0

Fetch from any agent

# Returns clean markdown by default
curl -X POST -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com"}' \
  http://your-machine:7411/fetch

# With options
curl -X POST -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","options":{"outputMode":"primary"}}' \
  http://your-machine:7411/fetch

This is how tools like openclaw get web content. POST a link, get text back, no browser management on the agent side.

Authenticated access

Content you already pay for

Set up a browser profile, log into your subscriptions once in a headed browser, and agent-fetch reuses those sessions. The profile stores cookies and auth state — not your full browser history. If the profile isn't configured, it fails fast instead of silently returning a login page.

One-time setup

# Log in interactively once
agent-browser \
  --profile ~/.agent-browser/profiles/work \
  --headed open https://stratechery.com

# Now agents can fetch paywalled content
curl -X POST -H 'Content-Type: application/json' \
  -d '{"url":"https://stratechery.com","options":{"strategyMode":"authenticated"}}' \
  http://localhost:7411/fetch

Output

Five ways to read a page

markdown Full-page content converted to markdown. The default.

primary Article extraction via Readability. Just the main content, no nav or sidebars.

html Cleaned rendered HTML with scripts, styles, and chrome stripped.

structured JSON with headings, sections, and links parsed from the page.

screenshot Full-page PNG via headless Chrome. For pages that resist text extraction.