Pagination is the most common reason scraping jobs fail silently. The first page works. The scraper looks successful. And you discover later that you only got 1% of the data you needed.
The problem isn’t just that pagination is complex — it’s that the same site often uses different pagination patterns on different pages. A product listing uses URL-based page numbers; the reviews use infinite scroll; the API uses cursor tokens. Each pattern needs a different approach.
This guide covers all four patterns. We’ll use UnWeb for Markdown conversion (which simplifies link extraction compared to raw HTML parsing) and Python for everything else.
The Four Pagination Patterns
| Pattern | Example | How to detect | Approach |
|---|---|---|---|
| URL page numbers | /products?page=2 | URL has page= or /page/2/ | Increment page param until empty |
| Next-page links | “Next →” link in footer | Markdown contains “Next” link | Follow next link recursively |
| Infinite scroll | News feeds, social-style lists | No next-page link, JS-rendered content | API reverse-engineer or Playwright |
| Cursor/offset API | ?after=cursor_xyz | Response contains next_cursor | Follow cursor until null |
Pattern 1: URL-Based Page Numbers
The simplest pattern. The URL has a page parameter (?page=2, ?p=3) or a path segment (/listings/page/2/). You increment the number until you get an empty page or a redirect.
import asyncio
import httpx
from typing import AsyncGenerator
UNWEB_API_KEY = "your_key"
async def fetch_markdown(url: str) -> str:
async with httpx.AsyncClient(timeout=30) as client:
resp = await client.get(
"https://api.unweb.info/v1/convert",
params={"url": url},
headers={"Authorization": f"Bearer {UNWEB_API_KEY}"}
)
resp.raise_for_status()
return resp.json()["markdown"]
async def scrape_paginated(
base_url: str,
page_param: str = "page",
max_pages: int = 100
) -> AsyncGenerator[str, None]:
"""
Yields Markdown content for each page until empty.
base_url: e.g. "https://example.com/products"
page_param: the query parameter name for page number
"""
for page_num in range(1, max_pages + 1):
url = f"{base_url}?{page_param}={page_num}"
markdown = await fetch_markdown(url)
# Stop if the page is empty or too short to have content
if len(markdown.strip()) < 200:
print(f"Page {page_num} appears empty. Stopping.")
break
print(f"Got page {page_num}: {len(markdown)} chars")
yield markdown
await asyncio.sleep(1) # Polite rate limiting
# Usage
async def main():
pages = []
async for page_markdown in scrape_paginated("https://example.com/products"):
pages.append(page_markdown)
print(f"Scraped {len(pages)} pages")
all_content = "\n\n---\n\n".join(pages)
Empty page detection: Don’t rely on HTTP status codes — many sites return 200 for empty pages. Check content length instead. A page with real listings will typically have 500+ characters of Markdown; an empty result page might have 100–200 characters (just nav and footer).
Pattern 2: Next-Page Links
Many sites don’t use page numbers — they use a “Next” link that points to the next page URL. This is more resilient than page numbers because you’re following the site’s own navigation rather than guessing URL patterns.
Since UnWeb converts pages to Markdown, we can extract the next-page link from the Markdown itself — no CSS selectors required. The key is knowing what the “Next” link looks like in Markdown ([Next →](url), [Next page](/page/2/), etc.).
import re
from urllib.parse import urljoin
def extract_next_link(markdown: str, base_url: str) -> str | None:
"""
Extract next-page URL from Markdown content.
Handles common patterns: [Next], [Next →], [Next page], [>], [»]
"""
patterns = [
r'\[(?:Next\s*[→›»>]?|Next\s+page|>|»)\]\(([^)]+)\)',
r'\[(?:next|Next)\]\(([^)]+)\)',
]
for pattern in patterns:
match = re.search(pattern, markdown, re.IGNORECASE)
if match:
href = match.group(1)
if href.startswith("http"):
return href
return urljoin(base_url, href)
return None
async def scrape_with_next_links(
start_url: str,
max_pages: int = 100
) -> list[str]:
pages = []
current_url = start_url
for _ in range(max_pages):
markdown = await fetch_markdown(current_url)
pages.append(markdown)
next_url = extract_next_link(markdown, current_url)
if not next_url:
print(f"No next link found after {len(pages)} pages.")
break
print(f"Following next link: {next_url}")
current_url = next_url
await asyncio.sleep(1)
return pages
Why Markdown beats HTML for link extraction: HTML next-page links are often deeply nested with classes like class="pagination__next btn btn--secondary". Markdown renders them as [Next →](url) — a simple regex pattern. No BeautifulSoup, no CSS selectors, no breakage on redesign.
Pattern 3: Infinite Scroll
Infinite scroll is the hardest pattern because the content loads via JavaScript as the user scrolls — there’s no next-page URL to follow. When you fetch the page with a standard HTTP request, you only get the initial above-the-fold content.
There are two approaches:
Option A: Reverse-engineer the underlying API
Infinite scroll almost always works by calling a hidden API endpoint behind the scenes. Open your browser’s DevTools → Network tab → scroll down → look for XHR/Fetch requests that return JSON with list items. These APIs are often simpler to call directly than the rendered page.
import httpx
import json
async def scrape_infinite_scroll_api(
api_url: str,
offset_param: str = "offset",
limit: int = 20,
max_items: int = 1000
) -> list[dict]:
"""
For sites that load infinite scroll via a JSON API with offset pagination.
Inspect Network tab to find the API endpoint and parameters.
"""
all_items = []
offset = 0
async with httpx.AsyncClient(
headers={"User-Agent": "Mozilla/5.0", "Accept": "application/json"},
timeout=30
) as client:
while len(all_items) < max_items:
resp = await client.get(
api_url,
params={offset_param: offset, "limit": limit}
)
resp.raise_for_status()
data = resp.json()
# Adjust these keys to match what the API actually returns
items = data.get("items") or data.get("results") or data.get("data", [])
if not items:
break
all_items.extend(items)
if len(items) < limit:
break # Last page had fewer items than requested
offset += limit
await asyncio.sleep(0.5)
return all_items
Option B: Playwright for true infinite scroll
When the API approach isn’t viable (heavily obfuscated, requires session tokens, etc.), you need a real browser. Playwright handles JavaScript execution and can simulate scrolling.
from playwright.async_api import async_playwright
async def scrape_infinite_scroll_playwright(
url: str,
scroll_pause: float = 1.5,
max_scrolls: int = 50
) -> str:
async with async_playwright() as p:
browser = await p.chromium.launch(headless=True)
page = await browser.new_page()
await page.goto(url, wait_until="networkidle")
prev_height = 0
scroll_count = 0
while scroll_count < max_scrolls:
# Scroll to bottom
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
await page.wait_for_timeout(int(scroll_pause * 1000))
new_height = await page.evaluate("document.body.scrollHeight")
if new_height == prev_height:
break # No more content loaded
prev_height = new_height
scroll_count += 1
html = await page.content()
await browser.close()
return html
# Then convert the full HTML to Markdown via UnWeb
# or parse directly with BeautifulSoup
Playwright note: Playwright is powerful but slow — 3–5 seconds per page vs. 0.5–1 second for a direct API call. For large-scale scraping, invest time in finding the underlying JSON API. Save Playwright for one-off tasks or sites with no viable API approach.
Pattern 4: Cursor-Based Pagination
Cursor-based pagination is common in modern APIs and some newer scraping targets. Instead of a page number, the API returns a next_cursor or after token that you pass to the next request. It’s more efficient for the server (no offset skipping) and handles real-time data better than page numbers.
async def scrape_cursor_paginated(
api_url: str,
cursor_response_key: str = "next_cursor",
cursor_param: str = "after",
max_pages: int = 200
) -> list[dict]:
all_items = []
cursor = None
async with httpx.AsyncClient(timeout=30) as client:
for _ in range(max_pages):
params = {"limit": 100}
if cursor:
params[cursor_param] = cursor
resp = await client.get(api_url, params=params)
resp.raise_for_status()
data = resp.json()
items = data.get("items") or data.get("data", [])
all_items.extend(items)
cursor = data.get(cursor_response_key)
if not cursor:
break # No more pages
await asyncio.sleep(0.25)
return all_items
Putting It Together: Auto-Detecting Pagination
For a robust scraper that handles multiple sites, it’s useful to auto-detect the pagination pattern from the first page.
async def detect_pagination_pattern(url: str) -> str:
"""
Returns: 'url_param', 'next_link', 'cursor', 'infinite_scroll', or 'single_page'
"""
markdown = await fetch_markdown(url)
# Check for explicit next-page link in Markdown
if re.search(r'\[(?:Next|next|>|»)\]', markdown):
return "next_link"
# Check URL for page parameter
if re.search(r'[?&]page=\d+', url) or re.search(r'/page/\d+/', url):
return "url_param"
# If content seems short, might be infinite scroll
if len(markdown) < 1000:
return "infinite_scroll"
return "single_page"
async def scrape_auto(url: str, max_pages: int = 100) -> list[str]:
pattern = await detect_pagination_pattern(url)
print(f"Detected pagination pattern: {pattern}")
if pattern == "next_link":
return await scrape_with_next_links(url, max_pages)
elif pattern == "url_param":
pages = []
async for page in scrape_paginated(url, max_pages=max_pages):
pages.append(page)
return pages
else:
# Single page or infinite scroll — handle separately
markdown = await fetch_markdown(url)
return [markdown]
Rate Limiting and Politeness
Pagination loops can make hundreds of requests to the same host. This will get you rate-limited or blocked without proper pacing.
import asyncio
import time
from collections import defaultdict
class RateLimiter:
def __init__(self, requests_per_second: float = 1.0):
self.min_interval = 1.0 / requests_per_second
self._last_request: dict[str, float] = defaultdict(float)
async def wait(self, host: str):
elapsed = time.monotonic() - self._last_request[host]
if elapsed < self.min_interval:
await asyncio.sleep(self.min_interval - elapsed)
self._last_request[host] = time.monotonic()
rate_limiter = RateLimiter(requests_per_second=1.0)
async def fetch_markdown_rate_limited(url: str) -> str:
from urllib.parse import urlparse
host = urlparse(url).netloc
await rate_limiter.wait(host)
return await fetch_markdown(url)
Reasonable defaults: 1 request/second is generally safe for most sites. For sites with a published API and rate limit header, respect the Retry-After header. For high-volume needs, check if the site offers a data API or dataset download — it’s better for both parties.
Error Handling and Resumability
Pagination jobs fail in the middle. A robust scraper saves progress and can resume from where it left off.
import sqlite3
import json
def init_progress_db(db_path: str = "scrape_progress.db"):
conn = sqlite3.connect(db_path)
conn.execute("""
CREATE TABLE IF NOT EXISTS pages (
url TEXT PRIMARY KEY,
markdown TEXT,
scraped_at TEXT
)
""")
conn.commit()
return conn
async def scrape_with_resume(
start_url: str,
db_path: str = "scrape_progress.db",
max_pages: int = 200
) -> list[str]:
conn = init_progress_db(db_path)
pages = []
current_url = start_url
for i in range(max_pages):
# Check if already scraped
existing = conn.execute(
"SELECT markdown FROM pages WHERE url = ?", (current_url,)
).fetchone()
if existing:
print(f"[cached] {current_url}")
markdown = existing[0]
else:
try:
markdown = await fetch_markdown_rate_limited(current_url)
conn.execute(
"INSERT OR REPLACE INTO pages (url, markdown, scraped_at) VALUES (?, ?, datetime('now'))",
(current_url, markdown)
)
conn.commit()
print(f"[scraped] page {i+1}: {current_url}")
except Exception as e:
print(f"[error] {current_url}: {e}")
break
pages.append(markdown)
next_url = extract_next_link(markdown, current_url)
if not next_url:
break
current_url = next_url
return pages
Which Pattern Should You Start With?
For a new scraping target:
- Open the site, look at the URL as you navigate to page 2. If it has
?page=2or/page/2/, use Pattern 1. - If no URL change but there’s a “Next” button, inspect it — most are simple
<a href="...">links. Use Pattern 2. - If no URL change and no visible next button, open DevTools Network tab, scroll down, and look for API calls that return JSON. Use Pattern 4 (cursor) or Pattern 3 option A.
- If the API is too complex to reverse-engineer, fall back to Playwright (Pattern 3 option B).
UnWeb: Clean Markdown From Any URL
Skip brittle HTML parsing. Get structured Markdown — quality-scored on every response — for any URL, including JS-rendered pages.