Structured Page Data for AI Agents

AI agents that interact with web pages need to understand what's on the page before they can do anything useful. The usual options don't work well. Send a screenshot to a vision model and it guesses where buttons are — usually wrong. Feed raw HTML to the LLM and you're burning tens of thousands of tokens on markup noise. Try the accessibility tree and you'll find it's broken on most real-world sites.

The Page Analysis API takes a different approach. It renders the URL in Chromium, executes all JavaScript, and returns structured JSON with every element on the page — headings, buttons, forms, links, images, navigation. Each element comes with its text, attributes, ARIA roles, href targets, and pixel-precise bounding box coordinates. One call, one response.

curl -X POST https://api.screenshotrun.com/v1/analyze \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://store.example.com/checkout",
    "include": ["interactive", "forms", "headings", "images", "navigation"]
  }'

Why do vision models fail at web page understanding?

The screenshot-to-action pipeline works like this: capture a screenshot, send it to GPT-4o or Claude, ask "where is the login button?", get back pixel coordinates, click there. Simple in theory. But in practice, bounding box accuracy is terrible.

Developers on OpenAI's own forums report that bounding box coordinates are wrong most of the time. Clicks land in the wrong place. Elements get misidentified. Dense layouts with small buttons confuse every model — GPT-4o, Gemini, Claude included. The accuracy just isn't there for production use.

It's slow, too. Every screenshot step sends an image through the vision model — a couple of seconds each time. Doesn't sound like much until your agent is doing twenty steps to complete a task. What should take ten seconds takes minutes. And the token bill grows with every image.

DOM-based structured data doesn't have these problems. The bounding box for a button comes from getBoundingClientRect() — the browser's own layout engine, not a model's guess. Coordinates are exact to the pixel. And the response takes about one second, not three.

The 7x token tax on raw HTML

Some agent frameworks skip vision entirely and just feed raw HTML to the LLM. That avoids the coordinate problem but creates a new one: token bloat. And it's a big one. Most of what's in a page's HTML — scripts, styles, nested divs, tracking pixels — is noise the model can't do anything useful with. The actual readable content is often a small fraction of the total. (And yes, some frameworks still dump everything by default.)

That means you're paying for tokens the model ignores, and the extra noise degrades reasoning quality too. The model spends attention on <div class="css-1dbjc4n r-1awozwy r-18u37iz"> instead of the actual content. Structured data strips away the markup and returns just the elements, text, and attributes — which can easily be an order of magnitude fewer tokens than raw HTML for the same page.

The website-to-markdown endpoint strips away the noise and returns clean text. But markdown alone loses something critical: spatial information. You know the page says "Add to Cart" but not where that button sits or what happens when you click it. The analyze endpoint gives you both — clean text content and element positions, attributes, and types.

Why the accessibility tree isn't the answer either

Playwright MCP and similar agent frameworks use the browser's accessibility tree as their primary page representation. It's compact, structured, and designed to describe page elements. On paper, ideal for agents.

On the real web, it's broken. The WebAIM Million report consistently finds that the vast majority of top websites have detectable WCAG failures. And here's the counterintuitive part — pages that actually try to use ARIA tend to have more errors than pages that don't, because most implementations get it wrong.

For agents, this means the accessibility tree on a typical production site is incomplete or misleading. Your agent works fine on well-built test sites, then falls apart on real targets. Buttons lack roles. Forms have no labels. Interactive elements are invisible to the accessibility API. The DOM doesn't have this problem — it shows what's actually rendered, regardless of ARIA quality.

Our analyze endpoint reads the rendered DOM, not the accessibility tree. It returns what's actually on the page — elements, text, attributes, coordinates — regardless of whether the site bothered to implement ARIA correctly.

Screenshot vs DOM vs markdown — which format for which task

Different agent tasks need different data formats. There's no single best representation for every job:

• Content extraction and summarization — markdown. Clean text, no noise, minimal tokens. Feed it straight to your LLM for analysis, summarization, or RAG pipelines.
• Form filling and data entry — form extraction. Every field with its name, type, placeholder, required status, and action URL. The agent knows exactly what to fill without parsing HTML.
• Navigation and clicking — clickable element extraction. Every button, link, and interactive element with bounding box coordinates. The agent clicks at exact positions, not vision model estimates.
• Page structure analysis — DOM-to-JSON. Full rendered DOM tree as structured data for custom parsing and deep inspection.
• Visual verification — screenshots with with_screenshot: true. When the agent needs to confirm that something looks right, not just that it exists in the DOM.

The analyze endpoint returns all of these from one API call. Your agent picks the format it needs for the current step — structured JSON for interaction, markdown for reading, screenshot for verification.

How agent integration works in practice

Here's a Python function that gives any LLM agent structured page understanding:

import requests

API_KEY = "YOUR_API_KEY"

def understand_page(url, needs_screenshot=False):
    """Give an agent structured understanding of any web page."""
    resp = requests.post(
        "https://api.screenshotrun.com/v1/analyze",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "url": url,
            "include": ["interactive", "forms", "headings", "images", "navigation"],
            "with_screenshot": needs_screenshot,
            "stealth": True,
        },
        timeout=60,
    )
    return resp.json()["result"]

# Agent gets the full page map
page = understand_page("https://store.example.com/product/123")

# Find the "Add to Cart" button — exact coordinates, not a vision model guess
cart_btn = next(
    (e for e in page["interactive"]
     if "cart" in e.get("text", "").lower() and e["tag"] in ("button", "a")),
    None
)

if cart_btn:
    x, y = cart_btn["rect"]["x"], cart_btn["rect"]["y"]
    print(f"Click 'Add to Cart' at ({x}, {y})")
    print(f"Element: <{cart_btn['tag']}>, disabled: {cart_btn.get('disabled', False)}")

# Check if there's a login form on the page
login_form = next(
    (f for f in page.get("forms", [])
     if any(fld.get("type") == "password" for fld in f.get("fields", []))),
    None
)

if login_form:
    print(f"Login form found: {login_form['method']} → {login_form['action']}")
    for field in login_form["fields"]:
        print(f"  {field['name']}: {field['type']}")

That call takes about one second. The agent gets exact button coordinates, form field schemas, navigation structure, heading hierarchy. Everything it needs to act on the page, without sending a screenshot through a vision model or stuffing raw HTML into the context window.

MCP integration for zero-code agent setup

The ScreenshotRun MCP Server exposes page analysis as a standard MCP tool. Any MCP-compatible agent — Claude, LangChain, CrewAI, AutoGen — can call it without custom code.

MCP has become the standard protocol for agent-tool interaction, with backing from Anthropic, OpenAI, Google, Microsoft, and AWS. But most web-related MCP servers — Firecrawl, Bright Data, Scrapfly — return markdown or raw HTML. None of them return structured element-level data with bounding boxes. So there's a gap, and the ScreenshotRun MCP server fills it.

{
  "tool": "screenshotrun_analyze",
  "input": {
    "url": "https://target.example.com",
    "include": ["interactive", "forms", "headings"],
    "stealth": true
  }
}

The agent calls one tool and gets back typed, structured JSON. You don't manage a browser process or maintain DOM parsing code. It works the same whether the agent runs in Claude Desktop, a LangChain pipeline, or a custom framework.

Handling protected and dynamic pages

Two problems that break most agent web access: bot detection and JavaScript rendering.

Sites behind Cloudflare, Akamai, or DataDome block headless browsers. Your agent gets a challenge page instead of the actual content. The stealth parameter handles this — the page sees a normal browser session, and your agent gets the real page.

SPAs and dynamic content are the second problem. A React app returns an empty shell until JavaScript runs. Lazy-loaded listings don't appear until someone scrolls. The analyze endpoint handles both — it renders in real Chromium, waits for JavaScript, and returns the fully rendered DOM instead of the raw HTML source that misses half the page.

Why fewer tokens means better results

The token savings from structured data aren't just about cost. They affect how well the agent reasons. When you feed raw HTML into an LLM, the context window fills up with markup that adds nothing — the model has to sort headings from tracking scripts from nested divs, all from the same stream of tags.

Structured JSON skips that entirely. Headings come labeled as headings. Buttons include their text, coordinates, and state. Forms list their field names and types. The result is typically an order of magnitude fewer tokens for the same page, and the model spends its attention on actual page content instead of figuring out what `

# Research agent: analyze multiple pages and extract key information
def research_topic(urls):
    findings = []
    for url in urls:
        page = understand_page(url)

        # Extract main content via headings
        headings = [h["text"] for h in page.get("headings", [])
                    if h.get("level") in (1, 2)]

        # Count interactive elements (complexity indicator)
        interactive = len(page.get("interactive", []))

        # Check for forms (data collection opportunities)
        forms = len(page.get("forms", []))

        findings.append({
            "url": url,
            "headings": headings,
            "interactive_elements": interactive,
            "forms": forms,
            "has_navigation": bool(page.get("navigation")),
        })

    return findings

# Agent researches competitor pricing pages
competitors = [
    "https://competitor-a.com/pricing",
    "https://competitor-b.com/pricing",
    "https://competitor-c.com/pricing",
]

report = research_topic(competitors)
for r in report:
    print(f"\n{r['url']}:")
    print(f"  Sections: {', '.join(r['headings'][:5])}")
    print(f"  Interactive elements: {r['interactive_elements']}")
    print(f"  Forms: {r['forms']}")