Vision models look at a screenshot and guess where the buttons are. They draw bounding boxes around things that look clickable, run OCR on the text inside, and return coordinates with varying confidence scores. GPT-4o gets about 5 out of 200 elements right on a complex page (per OpenAI's own developer forums). That's 2.5% accuracy. The model sees pixels, not DOM nodes, so it can't tell you what a button does — only where it appears to be.
The Page Analysis API reads the actual DOM. It loads your URL in Chromium, executes all JavaScript, waits for the page to stabilize, then walks the rendered DOM tree. Every element comes back with its tag name, text content, attributes, ARIA roles, href targets, event handlers, and a pixel-precise bounding rectangle. Not a guess. The browser's own layout engine computed those positions.
curl -X POST https://api.screenshotrun.com/v1/analyze \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://yoursite.com",
"include": ["interactive", "headings", "images", "forms", "navigation"]
}'What DOM-based detection gives you that vision can't
A vision model looking at a rendered page sees a blue rectangle with white text. It labels that "button" and draws a bounding box. But it can't tell you:
- Whether the element is a
<button>, an<a>styled as a button, or a<div>with an onclick handler - Where the link actually goes (the
hrefattribute) - Whether it has an ARIA label for screen readers
- Whether it's disabled, hidden behind a modal, or covered by another element
- What JavaScript runs when you click it
DOM-based detection knows all of this because it reads the source. The browser already parsed the HTML, applied the CSS, ran the scripts. We just extract what the browser already knows.
{
"interactive": [
{
"tag": "a",
"text": "Start Free Trial",
"href": "/register",
"rect": { "x": 920, "y": 340, "width": 180, "height": 48 },
"aria": { "role": "link" },
"classes": ["btn", "btn-primary"],
"visible": true
},
{
"tag": "button",
"text": "Add to Cart",
"type": "submit",
"rect": { "x": 640, "y": 1120, "width": 200, "height": 44 },
"aria": { "role": "button", "label": "Add Wireless Headphones to cart" },
"disabled": false,
"visible": true
}
]
}Every field in that response is a fact the browser computed, not a prediction a model made. You can assert against it, route automation logic through it, feed it to an AI agent as ground truth.
Why vision-based approaches keep failing at web pages
Computer vision works well on static documents — PDFs, scanned forms, mobile app screens with fixed layouts. Web pages are a different animal. A modern SPA renders content dynamically, loads elements on scroll, changes state on hover, and rearranges its layout across viewports. So vision models trained on static screenshots miss all of this.
Microsoft's OmniParser and Codia's VisualStruct both use screenshot-to-element pipelines. They take an image, segment it into regions, classify each region, and return coordinates. The pipeline breaks in predictable ways. Dropdown menus that haven't been opened yet are invisible. Elements inside iframes render as flat images. Tooltips and modals that appear on interaction don't exist in the screenshot. And anti-aliased text at different zoom levels produces different OCR results.
There's a deeper problem too. Two elements can look identical on screen but do completely different things. A "Submit" button in a payment form and a "Submit" button in a comment form are visually indistinguishable. DOM detection tells you one posts to /api/payments and the other to /api/comments. Vision detection tells you they're both rectangles with the word "Submit."
Building AI agents that interact with web pages
The AI agent market is projected at $12 billion in 2026, and most of these agents need to understand web pages. Browser-Use, Playwright MCP, and similar frameworks all hit the same bottleneck: figuring out what's on the page and what you can interact with.
The accessibility tree approach (used by some Playwright-based agents) sounds elegant but breaks on 95% of real websites. Most sites have incomplete ARIA markup, unlabeled buttons, divs styled as interactive elements without proper roles. The accessibility tree shows you what should be on the page according to ARIA specs. The DOM shows you what's actually there.
import requests
def get_page_elements(url, categories=None):
if categories is None:
categories = ["interactive", "headings", "images", "forms"]
resp = requests.post(
"https://api.screenshotrun.com/v1/analyze",
headers={"Authorization": f"Bearer {API_KEY}"},
json={"url": url, "include": categories},
timeout=60,
)
return resp.json()["result"]
# Agent discovers what's on the page
elements = get_page_elements("https://store.example.com/product/123")
# Find the "Add to Cart" button by text
cart_btn = next(
(e for e in elements["interactive"]
if "cart" in e.get("text", "").lower()),
None
)
if cart_btn:
# Agent now knows: exact coordinates, tag type, whether it's disabled
print(f"Button at ({cart_btn['rect']['x']}, {cart_btn['rect']['y']})")
print(f"Tag: {cart_btn['tag']}, Disabled: {cart_btn.get('disabled', False)}")
print(f"Form action: {cart_btn.get('form_action', 'N/A')}")That's the full loop: send a URL, get back structured data, make decisions. No Selenium, no Puppeteer, no browser instance running in your infrastructure. The agent gets a JSON map of the page and acts on it.
Replacing CSS selectors in automation scripts
CSS selectors are the duct tape of web automation. They work until the site changes a class name, restructures a div hierarchy, or switches from server-rendered to client-rendered markup. QA teams report spending 80% of their automation maintenance time on broken selectors.
Object detection by DOM analysis eliminates this. Instead of writing document.querySelector('.product-card .btn-primary') and hoping the class names don't change next sprint, you ask the API: "give me all interactive elements on this page." You get them all, with text, attributes, and coordinates. Your code finds elements by what they say and what they do, not by how they're nested in the HTML.
# Instead of brittle CSS selectors...
# driver.find_element(By.CSS_SELECTOR, ".hero-section .cta-btn.primary")
# ...describe what you're looking for
elements = get_page_elements("https://yoursite.com", ["interactive"])
# Find by visible text (survives any CSS refactor)
signup = next(
(e for e in elements["interactive"]
if e["tag"] in ("a", "button") and "sign up" in e.get("text", "").lower()),
None
)
# Find by destination (survives any text/design change)
pricing_link = next(
(e for e in elements["interactive"]
if e.get("href", "").endswith("/pricing")),
None
)These lookups survive CSS refactors, redesigns, framework migrations. The button text "Sign Up" and the link to /pricing are business logic. Class names are implementation details.
Competitive intelligence and monitoring
Tracking competitor pages is yet another area where DOM detection beats screenshots. A screenshot tells you the page changed visually. The DOM analysis tells you what changed: they added a new pricing tier, removed the "Enterprise" plan, changed the CTA text from "Book a Demo" to "Start Free Trial", or added three new product categories to the navigation.
# Track competitor changes structurally
yesterday = load_from_cache("competitor_state.json")
today = get_page_elements("https://competitor.com/pricing",
["headings", "interactive", "forms"])
# Detect new pricing tiers
old_headings = {h["text"] for h in yesterday.get("headings", [])}
new_headings = {h["text"] for h in today.get("headings", [])}
added = new_headings - old_headings
removed = old_headings - new_headings
if added or removed:
alert(f"Pricing page changed: +{added}, -{removed}")Run this on a webhook schedule and you get structural diffs of competitor pages without storing screenshot baselines or training a model to spot visual changes.
How does this compare to Firecrawl, Browserless, and JigsawStack?
Firecrawl scrapes pages and returns markdown or structured text. Good for content extraction, but it doesn't give you element coordinates or interactive element maps. You get the text of a page, not a model of its UI. If you need to know where a button sits on screen or what form fields a page has, Firecrawl won't help.
Browserless offers a /scrape endpoint that runs CSS selectors you provide against a rendered page. You have to know what you're looking for in advance — write the selectors, send them, get back the matching elements. No element discovery. So if you don't know the page structure, you're stuck. And you're paying for a full browser instance ($200+/month for dedicated plans) to run what amounts to querySelectorAll over the wire.
JigsawStack uses vision AI to detect UI elements from screenshots. Same problems as OmniParser: no attribute data, no href values, no ARIA roles, confidence scores instead of facts. Their pricing starts lower ($19/month) but the output is coordinates and labels, not structured DOM data.
Our analyze endpoint combines rendering, JavaScript execution, and DOM extraction in one call. You don't write selectors or train models, and you don't manage browser infrastructure. POST a URL, get back every element on the page with its full DOM context.
Accessibility auditing from the same response
Most accessibility scanners run their own browser session, parse the page separately, and return a different data format. With DOM-based object detection, the data you already have from the analyze call is enough for the most common checks. Every interactive element comes with its text, ARIA attributes, tag type, and role. Every image includes its alt attribute. And every form field includes its label, placeholder, and required status — no second tool needed.
result = get_page_elements("https://yoursite.com",
["interactive", "images", "forms"])
# Buttons without accessible names
unlabeled = [e for e in result["interactive"]
if e["tag"] == "button"
and not e.get("text", "").strip()
and not e.get("aria", {}).get("label")]
if unlabeled:
print(f"{len(unlabeled)} buttons have no accessible name")
# Images without alt text
no_alt = [img for img in result["images"] if not img.get("alt")]
if no_alt:
print(f"{len(no_alt)} images missing alt text")
# Form fields without labels
for form in result.get("forms", []):
for field in form.get("fields", []):
if not field.get("label") and not field.get("aria_label"):
print(f"Unlabeled field: {field['name']} ({field['type']})")Parameters for object detection
include— array of element categories:headings,interactive,forms,images,text,navigation. Request only what your use case needs.visible_only— defaulttrue. Set tofalseto include hidden elements, offscreen content, and elements behind modals.scope— CSS selector to limit detection to a specific container. Analyze just the header, sidebar, or product grid instead of the full page.device—desktop,mobile, ortablet. Same URL, different viewport, different element layout.delay— seconds to wait after page load before analyzing. Set to 2-3 for SPAs or pages with entrance animations.block_cookies,block_ads,block_chats— strip overlay elements before detection so GDPR dialogs and support pop-ups don't pollute your results.stealth— bypass bot detection on sites that block headless browsers.with_screenshot— capture a visual snapshot alongside the JSON for debugging or archival.
Full parameter documentation is on the Page Analysis docs page. Everything available for screenshots works here too: CSS/JS injection, proxy support, geolocation, timezone, cookies for authenticated sessions.
Why run object detection through ScreenshotRun
Vision-based tools give you probabilities. DOM-based detection gives you facts. The analyze endpoint renders the page in a real browser, walks the full DOM tree, and returns structured JSON with every element's tag, text, attributes, coordinates, and ARIA data. No model to train, no confidence thresholds to tune, and no OCR errors to filter.
You don't manage the browser infrastructure. We run Chromium instances, handle JavaScript execution, wait for dynamic content, bypass bot detection when needed. Your code makes one HTTP call and gets back a JSON map of the page. That works the same whether you're building an AI agent, automating QA, or monitoring competitor pages.
Screenshots, page analysis, markdown extraction, and PDF export all run on the same rendered page in a single request. One API key, one set of parameters, one Playground to test any URL before writing code. Twenty free requests per month to validate whether the output fits your pipeline.