Element Coordinates API — Get Bounding Boxes for Every Element on a Web Page

The Page Analysis API returns a bounding rectangle for every element it finds on a rendered web page. Four numbers per element: x position, y position, width, height — all in pixels, all from a real Chromium render. One POST to /v1/analyze and you get a coordinate map of an entire page without running a browser yourself or writing a single DOM query.

curl -X POST https://api.screenshotrun.com/v1/analyze \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "include": ["interactive", "headings"]
  }'

Every element in the response carries a rect array. A heading at [200, 80, 880, 48] starts 200px from the left edge, 80px from the top, spans 880px wide and 48px tall. A button at [920, 24, 160, 44] sits in the top-right corner. These aren't guesses — they come from the browser's own getBoundingClientRect(), rounded to the nearest pixel.

How bounding rectangles work under the hood

When you send a request, our renderer loads the URL in a headless Chromium instance. JavaScript executes, fonts load, CSS layouts resolve. Once the page stabilizes, we call getBoundingClientRect() on every discovered element and pack the result into four integers: [Math.round(x), Math.round(y), Math.round(width), Math.round(height)]. What comes back matches what you'd see in Chrome DevTools if you inspected each element manually.

Coordinates are relative to the document's top-left corner, not the viewport. A button 3,000px down a long page reports y: 3000. You'd need to scroll to reach it, but the API already knows where it is. This makes the data usable for full-page overlays, heatmap rendering, and automated click targeting without viewport offset math.

What comes back in the response

{
  "result": {
    "headings": [
      { "level": 1, "text": "Welcome to Acme", "rect": [200, 80, 880, 48] }
    ],
    "interactive": [
      { "tag": "a", "text": "Start Free Trial", "href": "/register", "role": "button", "rect": [920, 24, 160, 44] },
      { "tag": "button", "text": "Log In", "rect": [1090, 24, 80, 44] },
      { "tag": "input", "type": "email", "name": "email", "label": "Your email", "rect": [400, 320, 480, 44] }
    ],
    "images": [
      { "alt": "Product screenshot", "src": "/img/hero.webp", "rect": [100, 400, 1080, 600] }
    ],
    "stats": {
      "headings": 1,
      "interactive": 3,
      "images": 1
    }
  }
}

Six categories of elements can carry coordinates: headings, interactive elements (buttons, links, inputs), forms, images, text blocks, and navigation sections. Pick the ones you need with the include parameter — request ["interactive"] alone and the response drops to just clickable elements with their positions.

Why coordinates matter for AI agents

Any browser automation agent (LangChain, CrewAI, Claude MCP) needs to interact with web pages. Before it can click a button or fill a form, it needs to know where that button sits on screen. Raw HTML doesn't answer that — a <button> tag says nothing about its pixel position after CSS flexbox, media queries, and JavaScript layout shifts have done their work.

Accessibility trees weigh less than HTML but still lack spatial data. Screenshot-only approaches give visual context but cost extra vision tokens and lose structural detail. Bounding rectangles hit the sweet spot: each element gets a name, a type, and four integers that lock it to a pixel-precise spot. An agent can decide "click the Sign In button at [920, 24]" without parsing markup or running a vision model.

import requests

response = requests.post(
    "https://api.screenshotrun.com/v1/analyze",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "url": "https://app.example.com/dashboard",
        "include": ["interactive"],
        "stealth": True,
    },
    timeout=60,
)

data = response.json()

# Agent picks the target button by text, gets exact coordinates
for el in data["result"]["interactive"]:
    if el.get("text") == "Export Report":
        x, y, w, h = el["rect"]
        click_x = x + w // 2
        click_y = y + h // 2
        print(f"Click at ({click_x}, {click_y})")
        break

If you're using the ScreenshotRun MCP Server, Claude gets this coordinate map through the analyze_page tool. No browser infrastructure on your side, no Playwright scripts to maintain.

Visual regression testing with pixel coordinates

QA teams catch layout bugs by comparing element positions between deploys. Did the "Buy Now" button shift 20px down after the last CSS change? Is the hero image still above the fold on mobile? Bounding rectangles make these checks programmatic.

# Compare element positions between staging and production
staging = analyze("https://staging.example.com", ["interactive", "headings"])
production = analyze("https://example.com", ["interactive", "headings"])

for category in ["interactive", "headings"]:
    for s_el in staging["result"][category]:
        # Find matching element in production by text
        match = next(
            (p for p in production["result"][category]
             if p.get("text") == s_el.get("text")),
            None
        )
        if not match:
            print(f"MISSING in prod: {s_el.get('text')}")
            continue

        sx, sy, sw, sh = s_el["rect"]
        px, py, pw, ph = match["rect"]

        if abs(sy - py) > 10 or abs(sx - px) > 10:
            print(f"SHIFTED: '{s_el.get('text')}' moved from ({sx},{sy}) to ({px},{py})")

Screenshot diffing alone can't do this. A pixel-diff tool flags that something changed visually, but it won't tell you which element moved or by how much. Coordinate comparison gives you the name, old position, and new position — enough to file a precise bug report without opening a browser.

Above-the-fold detection

Knowing the viewport height lets you check if critical elements appear without scrolling. If you set a viewport of 800px tall, any element with y + height < 800 is above the fold.

const response = await fetch("https://api.screenshotrun.com/v1/analyze", {
  method: "POST",
  headers: {
    "Authorization": `Bearer ${process.env.SCREENSHOTRUN_KEY}`,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com",
    include: ["interactive", "headings"],
    height: 800,
  }),
}).then(r => r.json());

const viewport_h = 800;
const above = response.result.interactive.filter(
  el => el.rect[1] + el.rect[3] <= viewport_h
);
const below = response.result.interactive.filter(
  el => el.rect[1] + el.rect[3] > viewport_h
);

console.log(`${above.length} interactive elements above fold`);
console.log(`${below.length} require scrolling`);

// Flag if the primary CTA is below the fold
const cta = response.result.interactive.find(
  el => el.text?.toLowerCase().includes("sign up") || el.text?.toLowerCase().includes("get started")
);
if (cta && cta.rect[1] + cta.rect[3] > viewport_h) {
  console.warn(`WARNING: CTA "${cta.text}" is below the fold at y=${cta.rect[1]}`);
}

Run this across landing pages weekly and you'll catch when a design update pushes your signup button below the fold. Marketing teams use this data to validate that their highest-priority elements stay visible on first load.

Building click heatmap overlays

Combine a screenshot with element coordinates to generate annotated page overlays. Rectangles drawn at the exact positions of buttons, links, and forms create a visual element map that designers and product managers can review without inspecting the live site.

from PIL import Image, ImageDraw, ImageFont
import requests

# Get screenshot + analysis in one call
data = requests.post(
    "https://api.screenshotrun.com/v1/analyze",
    headers={"Authorization": f"Bearer {API_KEY}"},
    json={
        "url": "https://example.com",
        "include": ["interactive"],
        "with_screenshot": True,
    },
    timeout=60,
).json()

# Download the screenshot
img = Image.open(requests.get(data["meta"]["screenshot_url"], stream=True).raw)
draw = ImageDraw.Draw(img)

# Draw bounding boxes around interactive elements
colors = {"a": "#3B82F6", "button": "#EF4444", "input": "#10B981"}

for el in data["result"]["interactive"]:
    x, y, w, h = el["rect"]
    color = colors.get(el["tag"], "#F59E0B")
    draw.rectangle([x, y, x + w, y + h], outline=color, width=2)
    draw.text((x, y - 14), el.get("text", el["tag"])[:30], fill=color)

img.save("element-map.png")

Set with_screenshot: true in your request and the API returns both the structured coordinates and a rendered screenshot in a single call. Two outputs, two API credits, zero extra round trips.

Competitive monitoring: track layout changes

Watch how competitors position CTAs, navigation, and signup forms over time. Store coordinates from weekly analyses and compare them to spot A/B tests, redesigns, or changes meant to lift conversions.

When a competitor moves their pricing link from the header (y: 24) to a floating sidebar (y: 400), that's a signal. When their "Start Free Trial" button grows from 160px to 220px wide, they're testing conversions. Bounding rectangles capture layout-level changes like these. Content scrapers miss them entirely.

Pair this with brand monitoring to track both visual placement and content changes across competitor properties.

Accessibility auditing with element positions

Touch targets smaller than 44x44 pixels fail WCAG 2.5.8 guidelines. With bounding rectangles, you can flag failing targets across entire sites programmatically.

for el in data["result"]["interactive"]:
    x, y, w, h = el["rect"]
    if w < 44 or h < 44:
        print(f"WCAG fail: '{el.get('text', el['tag'])}' is {w}x{h}px (min 44x44)")
        print(f"  Position: ({x}, {y}), type: {el['tag']}")

Beyond touch targets, coordinates reveal overlapping clickables (two buttons sharing the same space), off-screen interactive elements (present in the DOM but unreachable), and focus order mismatches (tab order doesn't match visual order). Accessibility testing tools like axe-core check DOM properties. Coordinate data checks what users actually see and tap.

Scoping to a page section

Use the scope parameter with a CSS selector to limit coordinate output to one container. Analyzing a product card? Scope to .product-grid. Checking a checkout form? Scope to #checkout. Everything outside that container gets excluded from the response.

{
  "url": "https://shop.example.com/products",
  "scope": ".product-grid",
  "include": ["interactive", "images"]
}

Coordinates remain relative to the full document, not the scoped container. A button at [300, 1200, 120, 40] inside a scoped section is still 1,200px from the page top. This means you can combine scoped analyses with full-page screenshots and the positions line up perfectly.

Hidden elements and visibility filtering

By default, only visible elements get coordinates. An element hidden with display: none, visibility: hidden, or opacity: 0 gets skipped. Elements with zero width or height also get excluded — they exist in the DOM but occupy no space on screen.

Set visible_only: false to include everything. Hidden elements return coordinates of [0, 0, 0, 0] (the browser reports their bounding box as empty). This is useful for security audits — a hidden input with type="password" that occupies no screen space is worth investigating.

Pricing

Element coordinates come with every Page Analysis response at no extra cost — they're built into the rect field. Free and Starter plans include 20 analysis runs per month to test the workflow. Pro, Growth, and Business plans include unlimited runs. Each call uses 1 API credit (2 with with_screenshot: true). See the API docs for the full parameter reference.

FAQ#