Web scrapers parse raw HTML. That works fine for static sites built in 2015 — but a modern React dashboard, a Next.js marketing page, or a Vue product catalog doesn't have its content in the HTML source at all. Fetch the URL, look at what you get back: a <div id="root"></div>, a bundle script, and nothing else. The actual headings, paragraphs, and images exist only after JavaScript runs in a browser.
The Website to Markdown endpoint loads your URL in Chromium, executes JavaScript, waits for the DOM to populate, strips cookie banners and chat widgets, then converts the rendered page into markdown. You get back the page as a human would read it — not the source that a parser would see.
curl -X POST https://api.screenshotrun.com/v1/markdown \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/blog/post",
"block_cookies": true,
"block_ads": true
}'What the output looks like
The API returns markdown that preserves the page's content hierarchy without rendering artifacts. Headings keep their levels. Links retain their URLs. Code blocks stay formatted. Images get alt text and src attributes turned into markdown syntax. Everything else — tracking scripts, style blocks, SVG icons inlined as markup, hidden modals, consent dialogs — gets stripped.
{
"meta": {
"status": 200,
"took": 2.8,
"tokens": 1240,
"bytes": 4860,
"requests": 1
},
"result": {
"url": "https://example.com/blog/post",
"title": "How We Reduced Build Times by 60%",
"markdown": "# How We Reduced Build Times by 60%\n\nOur CI pipeline was taking 22 minutes per commit...\n\n## The bottleneck\n\nMost of the time went to dependency installation...\n\n## What we changed\n\n1. Switched to pnpm with a shared cache\n2. Parallelized test suites across 4 workers\n3. Moved lint checks to a pre-commit hook\n\n> The biggest win came from caching — build times dropped from 22 minutes to 8.\n\n## Results\n\n| Metric | Before | After |\n|--------|--------|-------|\n| Build time | 22 min | 8.5 min |\n| Test suite | 14 min | 4.2 min |\n\n",
"word_count": 842,
"links": 12,
"images": 3
}
}That response is about 5KB. The raw HTML for the same page? 120KB or more — packed with inline styles, script tags, tracking pixels, and DOM nodes that have nothing to do with the content. The markdown version is what an LLM or a human actually needs.
Rendered markdown vs. scraped markdown
The distinction matters more than it sounds. Tools like Firecrawl, Jina Reader, and various Apify actors convert HTML to markdown by parsing the raw HTTP response. That approach has three failure modes that show up constantly in production — and they're well-documented. Firecrawl's GitHub has open issues where pages return SCRAPE_ALL_ENGINES_FAILED because the HTML is a JS shell with no content. Jina Reader's issue tracker has reports of it returning empty results on even basic JavaScript-rendered test sites. These aren't obscure edge cases; they're the default behavior for any SPA.
First, client-side rendering. About 40% of the web runs on JavaScript frameworks now. A Next.js marketing site, a Nuxt blog, a SvelteKit documentation portal — these ship a minimal HTML shell and hydrate the content client-side. A raw HTTP fetch returns an empty page. Our renderer executes the JavaScript in Chromium and waits for the DOM to finish populating before touching anything. The markdown you get back includes headings, text, and images that only exist after React or Vue has mounted.
Second, cookie banners and overlays. Scraping tools convert what the HTML contains, including consent dialogs, promotional popups, and newsletter modals visible in the DOM from the first load. The rendered pipeline blocks these before the conversion step. Cookie banners get removed by our detection system (which targets z-index patterns above 9999 with position:fixed, plus known consent frameworks like OneTrust and Cookiebot). Chat widgets from Intercom, Drift, and Zendesk get stripped if you set block_chats: true. The output reflects the actual content, not the wrapper noise.
Third — and this one is surprisingly common — lazy-loaded content. Modern pages defer loading images, embeds, and below-the-fold sections until the user scrolls. A raw HTML parser sees data-src attributes instead of src, placeholder divs instead of real content, and Intersection Observer triggers that never fired. The renderer scrolls the page, waits for lazy content to load, and then converts. So the markdown includes images and sections that a static parser misses entirely.
Scoping to a specific section
Most pages have content you don't want in the markdown: navigation, footers, sidebars, related article widgets. The scope parameter takes a CSS selector and restricts the conversion to elements inside that container.
curl -X POST https://api.screenshotrun.com/v1/markdown \
-H "Authorization: Bearer YOUR_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"url": "https://example.com/docs/getting-started",
"scope": "article",
"block_cookies": true
}'With "scope": "article", only content inside the first <article> tag gets converted. Navigation, sidebars, footers, promo banners — excluded. That's the difference between getting a focused 800-word document and getting 3,000 words of noise wrapped around the content you actually wanted.
Which selectors work best? Depends on the site, but a few are nearly universal:
article— blog posts, news articles, documentation pagesmain— primary content area (most modern sites use this).post-contentor.entry-content— WordPress and similar CMS platforms#content— many custom builds[role="main"]— accessible sites that mark their main content region
Feeding web pages to LLMs
Large language models work well with markdown. It's compact, preserves structure, and every model parses it without special handling. But the quality of that markdown determines whether your AI pipeline produces useful output or hallucinated garbage.
Send raw HTML to GPT-4 or Claude and you burn 15,000–30,000 tokens on a single page — most of it scripts, styles, and DOM structure the model can't reason about. Send scraped markdown from a client-rendered site and you get an empty document. Send markdown with cookie banners still in it and your agent sees "We use cookies to improve your experience" as part of the page content. None of these are good.
The rendered pipeline solves all three. A typical blog post compresses from 120KB of raw HTML to 4-5KB of markdown — roughly 1,200 tokens instead of 25,000. That's a 95% reduction, and it preserves every paragraph, heading, link, and image reference the model needs for reasoning.
import requests
def get_page_markdown(url, scope=None):
payload = {
"url": url,
"block_cookies": True,
"block_ads": True,
"block_chats": True,
}
if scope:
payload["scope"] = scope
resp = requests.post(
"https://api.screenshotrun.com/v1/markdown",
headers={"Authorization": f"Bearer {API_KEY}"},
json=payload,
timeout=60,
)
return resp.json()["result"]
# Feed a web page to Claude as context
page = get_page_markdown("https://competitor.com/pricing", scope="main")
prompt = f"""Analyze this pricing page:
{page['markdown']}
Questions:
1. How many pricing tiers are there?
2. What's the cheapest paid plan?
3. Is there a free tier?
4. What limits does each tier have?"""An AI agent using the ScreenshotRun MCP Server can call get_markdown directly from Claude without wrapper code. The model receives text, reasons about the content, and responds — all within a manageable context window.
Building a content pipeline
Markdown output slots into any workflow that processes web content at scale. Content migration, competitive analysis, documentation aggregation, training data collection — all of these need text extracted from rendered pages, and all of them break when the extraction tool can't handle JavaScript.
urls = [
"https://blog-a.com/posts/intro-to-rag",
"https://blog-b.io/articles/vector-databases",
"https://docs-c.dev/guides/embeddings",
]
corpus = []
for url in urls:
result = get_page_markdown(url, scope="article")
corpus.append({
"url": result["url"],
"title": result["title"],
"content": result["markdown"],
"words": result["word_count"],
})
print(f" {result['title']} — {result['word_count']} words, {result['links']} links")
# corpus is now a list of clean markdown documents
# ready for embeddings, RAG indexing, or LLM summarizationThis works across any website without writing site-specific selectors. The renderer handles JavaScript, dynamic content loading, and noise removal. You write one pipeline and it processes sites built with React, WordPress, static HTML, Webflow, or anything else that runs in a browser.
Combining markdown with page analysis
The markdown endpoint and the Page Analysis endpoint do different things. Markdown gives you content as readable text. Analysis gives you page structure with pixel coordinates. When you need both — say, competitive intelligence where you want to read the copy AND know where each CTA sits — use them together.
# Get both markdown content and structured element data
markdown_result = get_page_markdown("https://competitor.com", scope="main")
analysis_result = get_page_analysis("https://competitor.com", include=["headings", "interactive", "images"])
# Markdown for content understanding
print(f"Content: {markdown_result['word_count']} words")
# Analysis for structural understanding
print(f"CTAs: {len([e for e in analysis_result['interactive'] if e.get('role') == 'button'])}")
print(f"Images: {analysis_result['stats']['images']}")
print(f"Navigation links: {sum(n['links'] for n in analysis_result.get('navigation', []))}")Markdown tells you what the page says. Element coordinates tell you where things sit. Together they give an AI agent enough context to understand and act on any web page — without sending raw HTML or processing a full screenshot through a vision model.
Content types that need special treatment
Documentation sites are the easiest case. Set scope: "main" to skip sidebars, enable cookie and chat blocking, and you're done. GitBook, Docusaurus, ReadMe — all of them wrap their docs in <main> or <article>, so the selector is predictable.
E-commerce product pages are trickier. Without scoping, the markdown includes every related product card, breadcrumb trail, and footer link on the page. Some pipelines want that. Most don't. Finding the right scope selector usually takes one look at DevTools — it's typically something like .product-detail or #product-description.
News articles are mostly straightforward (use <article>, set block_ads: true), but a fair number of news sites actively block headless browsers. Enable stealth mode for those. It patches the detection vectors — navigator.webdriver, Chrome DevTools protocol signals, missing plugin arrays — so the page renders normally.
SPAs behind auth gates need credentials. Pass cookies to authenticate, or use headers with a bearer token. The renderer loads the page in an authenticated state, JavaScript renders the protected content, and the markdown captures what a logged-in user would see. The screenshots behind login guide covers the auth patterns in detail.
Edge cases worth knowing about
No converter handles every page perfectly. Here's what to expect at the boundaries.
Infinite scroll pages (Twitter, Reddit, news aggregators) — the renderer scrolls a reasonable distance but can't scroll forever. You get the initial render plus about one viewport of additional content. Need more? Use the js parameter to trigger specific scroll behaviors or increase delay to give the feed time to load.
Tabs, accordions, and collapsible sections — these are common on FAQ pages and documentation. Whether collapsed content appears in the markdown depends on how the site built them. If the content is in the DOM but hidden with CSS (most cases), it shows up. If it's loaded via AJAX on click, it won't. You can use the js parameter to expand specific sections before the conversion runs.
Complex tables with colspan, rowspan, and nested headers get simplified. Markdown tables are inherently less expressive than HTML tables, so merged cells and nested structures lose some visual formatting. The text content is always preserved — you just might see a flatter table structure than the original.
Iframes — embedded content from other domains (YouTube, Twitter embeds, Codepen) appears as links, not inline content. Intentional. The markdown represents what's on the page, not what's loaded from third-party domains.
Key parameters
The markdown endpoint accepts the same browser configuration as Page Analysis and Screenshots. A few matter most for output quality:
scope— CSS selector to limit conversion to a specific container. Probably the most useful single parameter.block_cookies— removes consent banners (default:true)block_ads— removes ad network elementsblock_chats— removes Intercom, Drift, Zendesk, and similar widgetshide_selectors— array of CSS selectors to remove. For site-specific junk the built-in blockers miss.delay— seconds to wait after page load (0–10). Good for pages with delayed rendering.stealth— bypass bot detection on sites that block headless browserscss/js— inject custom styles or scripts before conversion. Expand accordions, dismiss modals, tweak the layout.device—desktop,mobile, ortablet. Mobile versions sometimes serve different content entirely.
How does this compare to other tools?
There are several ways to get markdown from a URL. Each makes different tradeoffs.
Firecrawl is a full web scraping platform — crawling, extraction, and markdown output. Handles JavaScript on its paid plans and returns good results. Built for bulk scraping and LLM training data at scale. If you also need structural page analysis, element coordinates, or screenshots alongside the markdown, you'd need a second tool.
Jina Reader has the simplest interface around — prepend r.jina.ai/ to any URL and get markdown back. Fast, free for light use. Where it struggles is JavaScript-heavy sites. The lighter rendering approach means complex SPAs can come back partial or empty. No cookie blocking, no ad removal, no scoping.
DIY with Puppeteer and Turndown is what most developers try first. Launch a browser, grab innerHTML, pipe it through the Turndown library. Works for simple pages, but then you need browser lifecycle management, proxy rotation, cookie consent removal, lazy image loading, and timeout handling. About 200 lines to get a basic version running, then a few months of edge-case fixes until it's production-grade.
ScreenshotRun sits between these. We run Chromium (like DIY Puppeteer), handle the infrastructure (like Firecrawl), and return scoped markdown that works on JS-heavy sites (better than Jina). What's different: we also do page analysis, screenshots, and PDF export from the same rendered page. One API, one platform, instead of stitching three services together.
Why use ScreenshotRun for this
The comparison above covers the technical tradeoffs. Here's the part that matters once you've picked a direction: what does the day-to-day actually look like?
You sign up, grab an API key, and hit the endpoint. No browser binaries to install, no Dockerfile to maintain, no Chromium version pinning when Google ships a breaking update (and they do — about every six weeks), no debugging why fonts render differently on your staging server. Your CI pipeline or AI agent makes an HTTP call and gets markdown back. If the site is slow to render, our infrastructure handles the wait and the retry. If Chromium crashes mid-render — which happens more often than people think on memory-heavy pages — the request gets rerouted automatically. You don't write that recovery logic; we already did.
The pricing model is straightforward. Each markdown conversion counts as one request toward your monthly quota, same as a screenshot or a PDF export. No separate metering for different output types. Twenty free requests per month are enough to validate output quality on your actual URLs before you commit to a paid plan — and the Playground lets you test live without writing any code at all.
For AI workflows specifically, the MCP Server removes the HTTP layer entirely. Claude calls get_markdown as a tool, gets text back, reasons about it. No wrapper functions, no request libraries. The model works with the content directly.