Python Web Scraping with Authentication

The gap between “scraping a public page” and “scraping login-gated content” trips up most developers who are new to scraping. The technical patterns aren’t complicated once you understand them, but they require a different mental model than what most tutorials cover.

This post covers the four authentication patterns you’ll encounter and how to handle each in Python using httpx for async HTTP and UnWeb for Markdown conversion once you’re authenticated.

Authorization matters: Only scrape content you’re authorized to access. Review the site’s Terms of Service. Scraping login-gated content without permission likely violates ToS and potentially applicable laws. The techniques in this post apply to your own accounts, authorized data collection, and sites that explicitly permit scraping.

The Four Authentication Patterns

Session cookies — site sets a cookie after login; subsequent requests include that cookie
Form-based login — POST to a login endpoint with username/password, get a session cookie back
Bearer token / OAuth — exchange credentials for a JWT or access token, include as Authorization: Bearer <token>
API key — static key passed as a header, query param, or in the request body

Pattern 1: Session Cookies (the Common Case)

The most common authentication pattern for traditional web apps. When you log in, the server sets a Set-Cookie header with a session ID. Subsequent requests that include that cookie are treated as authenticated.

The key insight: use an httpx.AsyncClient (or requests.Session) so cookies are automatically preserved across requests. Don’t try to manage cookies manually.

import httpx
import asyncio

LOGIN_URL = "https://example.com/login"
PROTECTED_URL = "https://example.com/dashboard"

async def scrape_with_session():
    async with httpx.AsyncClient(
        follow_redirects=True,
        timeout=30,
        headers={"User-Agent": "Mozilla/5.0 (compatible; my-scraper/1.0)"}
    ) as client:
        # Step 1: POST login credentials
        login_resp = await client.post(
            LOGIN_URL,
            data={"username": "your_email@example.com", "password": "your_password"},
        )
        login_resp.raise_for_status()

        # The client now holds the session cookie automatically
        # Step 2: access protected content
        protected_resp = await client.get(PROTECTED_URL)
        protected_resp.raise_for_status()

        return protected_resp.text

html = asyncio.run(scrape_with_session())
print(html[:500])

Why follow_redirects=True? Login endpoints frequently redirect after successful authentication (302 → dashboard). Without follow_redirects, you get the redirect response instead of the destination page.

Open DevTools → Network tab → log in manually → find the POST request to the login endpoint. Inspect the request payload to see the exact form field names (they vary: username / email / user, password / passwd / pass).

Some sites include a CSRF token in the login form that must be extracted from the login page HTML before submitting:

from bs4 import BeautifulSoup

async def login_with_csrf(client: httpx.AsyncClient, login_page_url: str,
                           post_url: str, email: str, password: str):
    # Get the login page to extract the CSRF token
    page = await client.get(login_page_url)
    soup = BeautifulSoup(page.text, "html.parser")

    # Find the CSRF token — name varies: _token, csrf_token, authenticity_token
    csrf = soup.find("input", {"name": "_token"})
    if not csrf:
        raise ValueError("CSRF token not found")

    await client.post(
        post_url,
        data={"email": email, "password": password, "_token": csrf["value"]}
    )

Pattern 2: Saving and Reusing Session Cookies

For scrapers that run repeatedly, logging in every time is slow and may trigger rate limiting. Save the session cookies to disk after the first login and reuse them until they expire.

import json
import os
from pathlib import Path

COOKIE_FILE = Path("session_cookies.json")

def save_cookies(client: httpx.AsyncClient):
    cookies = {name: value for name, value in client.cookies.items()}
    COOKIE_FILE.write_text(json.dumps(cookies))

def load_cookies() -> dict | None:
    if not COOKIE_FILE.exists():
        return None
    try:
        return json.loads(COOKIE_FILE.read_text())
    except Exception:
        return None

async def get_authenticated_client(login_url: str, email: str, password: str) -> httpx.AsyncClient:
    client = httpx.AsyncClient(follow_redirects=True, timeout=30)

    saved_cookies = load_cookies()
    if saved_cookies:
        for name, value in saved_cookies.items():
            client.cookies.set(name, value)
        # Quick check: try an authenticated request
        try:
            test = await client.get("https://example.com/api/me")
            if test.status_code == 200:
                return client  # Cookies still valid
        except Exception:
            pass

    # Cookies expired or don't exist — re-login
    await client.post(login_url, data={"email": email, "password": password})
    save_cookies(client)
    return client

Pattern 3: Bearer Token Authentication

Modern apps and APIs often use JWT or OAuth access tokens. You get a token from a login or token endpoint, then pass it as an Authorization header on every subsequent request.

import httpx
import time
import asyncio

TOKEN_URL = "https://api.example.com/auth/token"
API_BASE = "https://api.example.com/v1"

async def get_access_token(client_id: str, client_secret: str) -> tuple[str, float]:
    async with httpx.AsyncClient() as client:
        resp = await client.post(
            TOKEN_URL,
            json={"client_id": client_id, "client_secret": client_secret,
                  "grant_type": "client_credentials"}
        )
        resp.raise_for_status()
        data = resp.json()

    token = data["access_token"]
    # expires_in is seconds from now; store absolute expiry time
    expires_at = time.time() + data.get("expires_in", 3600) - 60  # 60s buffer
    return token, expires_at

class AuthenticatedClient:
    def __init__(self, client_id: str, client_secret: str):
        self.client_id = client_id
        self.client_secret = client_secret
        self._token: str | None = None
        self._expires_at: float = 0

    async def get_token(self) -> str:
        if not self._token or time.time() >= self._expires_at:
            self._token, self._expires_at = await get_access_token(
                self.client_id, self.client_secret
            )
        return self._token

    async def get(self, url: str, **kwargs) -> httpx.Response:
        token = await self.get_token()
        async with httpx.AsyncClient() as client:
            return await client.get(
                url,
                headers={"Authorization": f"Bearer {token}"},
                **kwargs
            )

# Usage
async def main():
    auth_client = AuthenticatedClient("my_client_id", "my_client_secret")
    resp = await auth_client.get(f"{API_BASE}/users/me")
    print(resp.json())

Pattern 4: API Key Authentication

The simplest pattern. A static key is passed as a header, query parameter, or in a custom header. The key doesn’t expire unless rotated manually.

import httpx
import os

API_KEY = os.environ["MY_API_KEY"]  # Never hardcode keys in source

# Pattern A: Header (most common)
async def api_key_header(url: str) -> dict:
    async with httpx.AsyncClient() as client:
        resp = await client.get(url, headers={"X-API-Key": API_KEY})
        resp.raise_for_status()
        return resp.json()

# Pattern B: Query parameter
async def api_key_query(url: str) -> dict:
    async with httpx.AsyncClient() as client:
        resp = await client.get(url, params={"api_key": API_KEY})
        resp.raise_for_status()
        return resp.json()

# Pattern C: Bearer token (when key is used as a bearer)
async def api_key_bearer(url: str) -> dict:
    async with httpx.AsyncClient() as client:
        resp = await client.get(
            url,
            headers={"Authorization": f"Bearer {API_KEY}"}
        )
        resp.raise_for_status()
        return resp.json()

Combining Auth with UnWeb for Clean Markdown

Once you have an authenticated session, you can pass those cookies to the UnWeb API to convert login-gated pages to Markdown. UnWeb accepts a cookies parameter for this purpose:

import httpx

UNWEB_API_KEY = "your_unweb_key"

async def fetch_authenticated_markdown(
    url: str,
    session_cookies: dict
) -> str:
    async with httpx.AsyncClient(timeout=30) as client:
        resp = await client.post(
            "https://api.unweb.info/v1/convert",
            headers={"Authorization": f"Bearer {UNWEB_API_KEY}"},
            json={
                "url": url,
                "cookies": session_cookies  # Pass your authenticated cookies
            }
        )
        resp.raise_for_status()
        return resp.json()["markdown"]

# Usage with the session pattern
async def main():
    async with httpx.AsyncClient(follow_redirects=True) as client:
        await client.post(
            "https://example.com/login",
            data={"email": "you@example.com", "password": "your_pass"}
        )
        # Extract cookies as dict
        cookies = dict(client.cookies)

    # Now fetch login-gated content as clean Markdown
    markdown = await fetch_authenticated_markdown(
        "https://example.com/members/dashboard",
        cookies
    )
    print(markdown[:500])

Common Authentication Debugging Tips

Getting 401/403 after login: Check that cookies are being sent. Log client.cookies after login to verify the session cookie is present.
Login succeeds but redirects to wrong page: The login POST may redirect to a page that checks for a completed profile or 2FA. Follow the redirect chain manually in DevTools to see what’s happening.
Getting blocked: Sites detect non-browser clients via missing headers (User-Agent, Accept-Language, Referer). Add realistic browser headers and consider adding small random delays.
CSRF token mismatch: The CSRF token must match the current session. Always fetch the login page first, extract the token, and submit it in the same session.
Token expiry mid-scrape: Use the AuthenticatedClient pattern above to auto-refresh tokens before they expire.

UnWeb: Markdown Conversion for Authenticated Pages

Pass your session cookies and get clean Markdown back from login-gated content — no HTML parsing, no CSS selectors.

Get started free