You need data from a website that doesn’t have an API. Your options are traditionally: write a scraper with CSS selectors and XPath, pay for a commercial data provider, or give up.
The CSS selector approach works until the site redesigns. Then you spend an hour fixing selectors, ship the fix, and repeat the cycle every few months. The commercial provider works but costs $100–500/month and may not support the specific site you need.
There is a better option: use an LLM to extract structured data from the page’s text content. No selectors, no XPath, no DOM traversal. The extraction logic lives in a schema definition and a prompt — both of which survive layout changes entirely.
This tutorial builds a complete JSON API wrapper for any website: clean Markdown fetch via UnWeb, typed extraction with Claude, served through FastAPI with a cache layer. Total code: under 80 lines.
The Architecture
The pipeline has three stages:
- Fetch: Convert the URL to clean Markdown using UnWeb. This strips navigation, ads, cookie banners, and HTML structure — leaving just the content, 10–40x smaller than raw HTML.
- Extract: Send the Markdown to Claude with a Pydantic schema as the output specification. Claude reads the text and returns structured JSON matching the schema.
- Serve: Wrap the pipeline in a FastAPI endpoint with a simple cache so repeated requests don’t burn API credits.
Prerequisites: Python 3.11+, pip install fastapi uvicorn anthropic httpx pydantic, an UnWeb API key (app.unweb.info), and an Anthropic API key.
Step 1: Fetch Clean Markdown
Raw HTML is 15,000–40,000 tokens for a typical page. After stripping navigation, scripts, styles, and boilerplate, the content is usually 1,000–4,000 tokens — a 10x+ reduction. That reduction is important for both cost and accuracy: LLMs extract data more reliably from clean text than from noisy HTML.
import httpx
import os
UNWEB_API_KEY = os.environ["UNWEB_API_KEY"]
UNWEB_BASE = "https://api.unweb.info/v1"
async def fetch_markdown(url: str) -> str:
"""Fetch a URL and return clean Markdown content."""
async with httpx.AsyncClient(timeout=30) as client:
response = await client.get(
f"{UNWEB_BASE}/convert",
params={"url": url, "format": "markdown"},
headers={"Authorization": f"Bearer {UNWEB_API_KEY}"},
)
response.raise_for_status()
data = response.json()
content = data.get("markdown", "")
if len(content) < 100:
raise ValueError(f"Page content too short — likely a bot block or empty page")
return content
Step 2: Extract Structured Data with Claude
Define what you want as a Pydantic model. Claude reads the Markdown and fills in the schema — it handles missing fields gracefully (returns None), normalizes inconsistent formatting, and works across different page layouts as long as the underlying content is present.
import anthropic
import json
from pydantic import BaseModel
from typing import Optional
ANTHROPIC_API_KEY = os.environ["ANTHROPIC_API_KEY"]
class ProductData(BaseModel):
name: str
price: Optional[float] = None
currency: Optional[str] = None
availability: Optional[str] = None
rating: Optional[float] = None
review_count: Optional[int] = None
description: Optional[str] = None
sku: Optional[str] = None
async def extract_data(markdown: str, schema: type[BaseModel]) -> dict:
"""Extract structured data from Markdown using Claude."""
schema_json = schema.model_json_schema()
client = anthropic.Anthropic(api_key=ANTHROPIC_API_KEY)
message = client.messages.create(
model="claude-haiku-4-5-20251001", # fast and cheap for extraction
max_tokens=1024,
messages=[{
"role": "user",
"content": f"""Extract structured data from the following web page content.
Return a JSON object matching this schema exactly:
{json.dumps(schema_json, indent=2)}
Rules:
- Return only the JSON object, no explanation
- Use null for fields that aren't present in the content
- Normalize prices to float (remove currency symbols)
- For currency, extract the 3-letter code (USD, EUR, GBP) if present
Web page content:
{markdown[:8000]}"""
}]
)
raw = message.content[0].text.strip()
# Strip markdown code fences if Claude wraps the JSON
if raw.startswith("```"):
raw = raw.split("```")[1]
if raw.startswith("json"):
raw = raw[4:]
return json.loads(raw.strip())
Step 3: FastAPI Endpoint with Cache
Wrap the pipeline in a FastAPI endpoint. A simple in-memory cache with a TTL prevents redundant fetches — product pages don’t change by the second, so a 15-minute cache is usually fine. For higher traffic, swap the dict for Redis.
from fastapi import FastAPI, HTTPException, Query
from datetime import datetime, timedelta
import asyncio
app = FastAPI(title="Website JSON API")
# Simple in-memory cache: {url: (data, expires_at)}
_cache: dict[str, tuple[dict, datetime]] = {}
CACHE_TTL = timedelta(minutes=15)
@app.get("/api/product", response_model=ProductData)
async def get_product(url: str = Query(..., description="Product page URL to extract data from")):
"""Extract product data from any product page URL."""
# Check cache
if url in _cache:
data, expires_at = _cache[url]
if datetime.utcnow() < expires_at:
return data
try:
markdown = await fetch_markdown(url)
extracted = await extract_data(markdown, ProductData)
result = ProductData(**extracted)
except ValueError as e:
raise HTTPException(status_code=422, detail=str(e))
except Exception as e:
raise HTTPException(status_code=500, detail=f"Extraction failed: {str(e)}")
# Store in cache
_cache[url] = (result.model_dump(), datetime.utcnow() + CACHE_TTL)
return result
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000)
Run it with python main.py and call it:
curl "http://localhost:8000/api/product?url=https://example-store.com/products/widget-pro"
{
"name": "Widget Pro",
"price": 49.99,
"currency": "USD",
"availability": "In Stock",
"rating": 4.7,
"review_count": 312,
"description": "The Widget Pro handles all standard widget tasks with 2x the throughput of the original Widget.",
"sku": "WGT-PRO-001"
}
Making It Generic
The ProductData schema is specific to product pages. The same pattern works for any structured content by swapping the schema. Examples:
class JobPosting(BaseModel):
title: str
company: str
location: Optional[str] = None
salary_min: Optional[int] = None
salary_max: Optional[int] = None
remote: Optional[bool] = None
posted_date: Optional[str] = None
requirements: list[str] = []
class RestaurantInfo(BaseModel):
name: str
cuisine: Optional[str] = None
price_range: Optional[str] = None # "$", "$$", "$$$"
rating: Optional[float] = None
address: Optional[str] = None
phone: Optional[str] = None
hours: Optional[dict] = None
class NewsArticle(BaseModel):
headline: str
author: Optional[str] = None
published_at: Optional[str] = None
summary: Optional[str] = None
tags: list[str] = []
To support multiple schemas through a single endpoint, accept the schema name as a parameter and dispatch to the right model:
SCHEMAS = {
"product": ProductData,
"job": JobPosting,
"restaurant": RestaurantInfo,
"article": NewsArticle,
}
@app.get("/api/extract")
async def extract_any(
url: str = Query(...),
schema: str = Query("product", description="Schema to extract: product, job, restaurant, article")
):
if schema not in SCHEMAS:
raise HTTPException(status_code=400, detail=f"Unknown schema. Options: {list(SCHEMAS.keys())}")
schema_class = SCHEMAS[schema]
markdown = await fetch_markdown(url)
extracted = await extract_data(markdown, schema_class)
return schema_class(**extracted).model_dump()
How It Handles Layout Changes
This is where the LLM approach wins decisively. Consider what happens when a site redesigns:
| Change | CSS Selector Scraper | LLM Extraction |
|---|---|---|
Price moved from .product-price to .price-wrapper .current | Breaks — selector returns null | Works — Claude reads “Price: $49.99” regardless of selector |
| Site adds currency symbol styling | Breaks — raw text includes <sup> tags | Works — Markdown strips HTML, Claude normalizes to float |
| Rating moved from stars to text “4.7 out of 5” | Breaks — scraper expected star count | Works — Claude understands “4.7 out of 5” = 4.7 |
| Site adds country-specific pricing variants | Depends — may need selector update | Works — Claude picks the relevant price for context |
The CSS selector scraper needs maintenance every time a site updates. The LLM extractor needs maintenance when the content changes in a way that breaks schema assumptions — which is rare and usually intentional (like a site removing a feature entirely).
Cost and Performance
For product pages:
- UnWeb fetch: ~100ms, fraction of a cent per call
- Claude Haiku extraction: ~500ms, ~$0.0005–0.002 per call depending on page length
- Total per uncached request: ~600ms, ~$0.001–0.003
With a 15-minute cache, a pipeline monitoring 100 product pages every 15 minutes costs roughly $0.10–0.30/day. For most use cases — price monitoring, inventory tracking, content aggregation — that is well within acceptable range.
Token optimization: UnWeb’s format=markdown returns the full page content. For pages with large content sections you don’t need (like long descriptions or user reviews), add a max_length parameter or truncate the Markdown before passing it to Claude. The extraction schema fields you care about are usually in the first 2,000 tokens of the page.
Production Considerations
Replace in-memory cache with Redis if you’re running multiple workers or need persistence across restarts. pip install redis and use redis.asyncio.
Add rate limiting before production use. Both UnWeb and the Anthropic API have rate limits — wrap calls with a semaphore or use asyncio.Semaphore to limit concurrency.
Add extraction validation for critical fields. If price is None on a product page where it should never be None, log it and alert rather than silently returning null.
Handle bot blocks. Some sites block automated requests. UnWeb handles many common patterns (JavaScript rendering, cookie acceptance), but high-value targets may require additional handling. Check the content_length in the UnWeb response — a very short response usually means the page returned a bot block rather than content.
UnWeb: Clean Markdown from any URL
Get the clean text your LLM pipelines need. No JavaScript rendering setup, no proxy management, no bot-block handling. One API call returns structured Markdown from any public URL.