DOM to JSON API — Convert Any Rendered Web Page to Structured JSON

Raw HTML is a terrible data format. A typical web page is 80,000–200,000 characters of nested divs, inline scripts, style blocks, tracking pixels, and the actual content buried somewhere in the middle. Parsing it yourself means writing CSS selectors that break when the site changes its markup, handling malformed tags, and dealing with the fact that half the page probably didn't exist in the HTML at all — it was rendered by JavaScript after the browser loaded. The analyze endpoint skips all of that. Send a URL, get back clean JSON organized into six categories, each element tagged with its pixel coordinates via bounding rectangles.

curl -X POST https://api.screenshotrun.com/v1/analyze \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "include": ["headings", "interactive", "forms", "images", "text", "navigation"]
  }'

The response is flat and predictable. No nested DOM tree to traverse, no sibling/child relationships to reconstruct. Each category is an array of objects with consistent fields. Headings have a level and text. Interactive elements have a tag, text, and conditional fields like href or role. Forms have an action URL, method, and field count. Every object includes a rect — four integers for pixel position.

What the JSON looks like

Here's an abbreviated response for a SaaS marketing page. The real output would be longer, but the structure stays the same regardless of how complex the source HTML is.

{
  "result": {
    "stats": {
      "headings": 5,
      "interactive": 18,
      "forms": 1,
      "images": 6,
      "text": 9,
      "navigation": 2
    },
    "headings": [
      { "level": 1, "text": "Build better products", "rect": [240, 140, 800, 52] },
      { "level": 2, "text": "Features", "rect": [240, 920, 800, 36] },
      { "level": 2, "text": "Testimonials", "rect": [240, 2200, 800, 36] }
    ],
    "interactive": [
      { "tag": "a", "text": "Docs", "href": "/docs", "rect": [600, 20, 48, 32] },
      { "tag": "button", "text": "Start Free Trial", "rect": [960, 16, 160, 44] },
      { "tag": "input", "type": "email", "name": "email", "label": "Work email", "rect": [360, 280, 400, 48] }
    ],
    "forms": [
      { "action": "/signup", "method": "POST", "fields": 2, "rect": [360, 260, 560, 120] }
    ],
    "images": [
      { "alt": "Dashboard preview", "src": "/images/hero-dashboard.webp", "rect": [120, 400, 1040, 520] }
    ],
    "text": [
      { "tag": "p", "text": "Ship faster with real-time collaboration, automated testing, and one-click deploys.", "rect": [240, 200, 800, 48] }
    ],
    "navigation": [
      { "label": "Main menu", "links": 6, "rect": [0, 0, 1280, 64] },
      { "label": "", "links": 14, "rect": [0, 3800, 1280, 240] }
    ],
    "page": {
      "title": "Acme — Build better products",
      "url": "https://acme.com/",
      "width": 1280,
      "height": 4040
    }
  }
}

That entire response weighs about 2KB. The raw HTML for the same page would be 150KB or more — 75 times larger, with none of the structure.

Why raw HTML fails for data extraction

Three problems make raw HTML unreliable as a data source. First, JavaScript-rendered content is invisible. A React dashboard, a Vue product page, an Angular admin panel — fetch their URLs with curl and you get a skeleton: a <div id="root"></div> and a pile of script tags. The actual headings, buttons, and images don't exist until the browser runs those scripts. Static parsing never sees them.

Second, HTML is noisy. For every paragraph of real text, there are tracking scripts, SVG icons inlined as markup, style blocks, hidden modals that won't display until a user clicks something, and noscript fallbacks that confuse any regex-based parser. Extracting "just the buttons" from raw HTML requires knowing the site's specific markup conventions, and those conventions change without warning.

Third, HTML carries no spatial data. You know a button exists, but you don't know if it's at the top of the page or buried below six screens of content. You know an image is in the DOM, but not whether it's 40px wide (an icon) or 1200px wide (a hero banner). The analyze endpoint gives you coordinates because it renders the page first. It measures elements after CSS has been applied and layout has been calculated.

Feeding structured page data to an LLM

Large language models have context windows measured in tokens. Sending 80,000 characters of raw HTML to GPT-4 or Claude burns through 20,000–30,000 tokens just to describe the page, leaving little room for the agent's instructions and reasoning. Most of those tokens represent boilerplate — navigation markup, footer links, script tags the model can't act on anyway.

The analyze endpoint compresses a page into what matters. A typical response for a medium-complexity page is 1,500–3,000 tokens of structured JSON. That's a 90–95% reduction that preserves the information an agent actually needs: what's on the page, what each thing says, and where it sits.

import requests

def get_page_json(url, categories=None):
    payload = {"url": url}
    if categories:
        payload["include"] = categories

    resp = requests.post(
        "https://api.screenshotrun.com/v1/analyze",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json=payload,
        timeout=60,
    )
    return resp.json()["result"]

# Give an LLM only the structured data it needs
page = get_page_json("https://competitor.com", ["headings", "interactive"])

prompt = f"""Analyze this page structure:
- Headings: {page['headings']}
- Interactive elements: {page['interactive']}
- Stats: {page['stats']}

What type of page is this? What's the main call to action?"""

An agent using the ScreenshotRun MCP Server gets this data through analyze_page without wrapper code. Claude reads the JSON, reasons about the page structure, and decides its next action — all within a compact context window.

Building a web scraping pipeline

Traditional scraping with Beautiful Soup or Cheerio means writing selectors tailored to one site's markup. When the site redesigns, your selectors break. The analyze endpoint abstracts away the HTML layer entirely. You ask for "headings" and get headings, regardless of whether the site wraps them in <header>, <div class="title">, or a React component that renders at runtime.

urls = [
    "https://site-a.com/blog/post-1",
    "https://site-b.com/articles/intro",
    "https://site-c.io/news/update",
]

for url in urls:
    data = get_page_json(url, ["headings", "text", "images"])

    title = data["headings"][0]["text"] if data["headings"] else "Untitled"
    body = " ".join(block["text"] for block in data["text"])
    image_count = data["stats"]["images"]

    print(f"{title}")
    print(f"  {len(body)} chars of text, {image_count} images")
    print(f"  Source: {url}")

This works across any website because the output format is site-independent. You never touch a CSS selector. The renderer handles JavaScript execution, cookie banners, lazy-loaded content, and everything else that makes raw HTML parsing fragile.

Scoping to a specific section

Sometimes you want only part of a page — the main content area, a sidebar, or a specific widget. The scope parameter takes a CSS selector and restricts the scan to elements within that container.

curl -X POST https://api.screenshotrun.com/v1/analyze \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/blog/post",
    "include": ["headings", "text", "images"],
    "scope": "article"
  }'

With "scope": "article", the API returns only elements inside the first <article> tag. Navigation, footer, sidebar — all excluded from the response. This trims the JSON down to exactly what you need, and it handles the common case where you want blog content without chrome.

Monitoring pages for structural changes

Store analysis results and diff them over time. When a competitor adds a new section, changes their navigation, or moves a CTA, you catch it automatically.

import json
from pathlib import Path

def snapshot_page(url, snapshot_dir="./snapshots"):
    data = get_page_json(url)
    path = Path(snapshot_dir) / f"{url.replace('/', '_')}.json"

    old = json.loads(path.read_text()) if path.exists() else None
    path.write_text(json.dumps(data, indent=2))

    if old is None:
        return {"status": "first_snapshot"}

    changes = []

    # Heading count changed?
    if old["stats"].get("headings") != data["stats"].get("headings"):
        changes.append(f"Headings: {old['stats']['headings']} → {data['stats']['headings']}")

    # Navigation link count changed?
    for i, (o, n) in enumerate(zip(old.get("navigation", []), data.get("navigation", []))):
        if o["links"] != n["links"]:
            changes.append(f"Nav #{i}: {o['links']} → {n['links']} links")

    # New interactive elements?
    old_count = old["stats"].get("interactive", 0)
    new_count = data["stats"].get("interactive", 0)
    if abs(new_count - old_count) > 3:
        changes.append(f"Interactive elements: {old_count} → {new_count}")

    return {"status": "changed" if changes else "unchanged", "changes": changes}

Run this daily with scheduled screenshots and you get a lightweight monitoring system. No pixel comparison needed — you're diffing structured data, so false positives from ad rotations or dynamic content are eliminated.

Visible versus hidden elements

By default, the API returns only visible elements. An element is visible when its computed display is not none, visibility is not hidden, opacity is not 0, and it has non-zero dimensions. This filters out script-injected tracking pixels, off-screen modals, and collapsed accordion content that would clutter the response.

Set visible_only: false when you need the full picture. Security scanning is one reason — hidden elements on a compromised page might include invisible links or injected forms that users can't see but bots can follow. QA testing is another: you might want to verify that a modal's content is present in the DOM before the trigger click reveals it.

Choosing which categories to request

Requesting all six categories for every call works, but it's heavier than necessary for focused tasks. The include parameter lets you pick what you need.

A few practical combinations:

• ["headings"] alone is enough for SEO heading audits across an entire site
• ["interactive"] gives an AI agent a map of every button, link, and input on the page
• ["headings", "interactive", "images"] covers most competitive analysis needs
• ["forms"] is what you want for login page detection or checkout flow mapping
• Omit include entirely and you get all six — useful for one-off exploration but larger payloads

Smaller requests mean faster responses. If you only need heading structure, asking for ["headings"] skips the DOM queries for images, forms, text blocks, and everything else.

Handling JavaScript-heavy sites

Single-page applications built with React, Vue, Angular, or Svelte generate their DOM at runtime. A static HTTP fetch returns the shell — usually an empty div and a bundle script. Our renderer runs that JavaScript in Chromium and waits for the DOM to populate before scanning it. The JSON you get back reflects the fully rendered page, not the server's initial HTML response.

For sites that actively block headless browsers (checking navigator.webdriver, measuring mouse events, fingerprinting Chrome DevTools protocol), enable stealth mode. It patches the common detection vectors so the page renders as it would for a regular visitor. Combine it with cookie blocking and ad blocking to get clean JSON without consent banners or promotional overlays muddying the results.

What you'd build yourself (and why you probably shouldn't)

Replicating this with Puppeteer or Playwright takes about 200 lines of JavaScript: launch a browser, navigate to the URL, wait for network idle, query the DOM for each element category, call getBoundingClientRect() on every match, filter by visibility, serialize to JSON, close the browser, handle crashes. Then you need a server to run it, a queue for concurrent requests, proxy rotation for sites that block datacenter IPs, and someone to maintain it when Chromium updates break your selectors.

Text extractors like Firecrawl and Jina Reader solve a different problem. They return markdown or cleaned prose — useful for feeding documents into a RAG pipeline, but they strip spatial data and element types in the process. You get the text of a page; you don't get its structure.

The analyze endpoint sits between these two approaches. It runs a real browser (like DIY Puppeteer) but returns pre-categorized JSON (like a scraping API), with pixel coordinates that neither approach typically provides. Test it in the Playground with any URL, or jump straight to the API reference for integration code.

FAQ#