Web Scraping Python

User-Agents and Browser Fingerprinting

· 1 min read

The default requests User-Agent (python-requests/2.x) is the fastest possible way to get blocked. But modern anti-bot stacks look at far more than one header.

The full fingerprint, roughly in order of what sites actually check:

  1. User-Agent string — easiest to fix
  2. Accept / Accept-Language / Accept-Encoding — browsers send these; requests minimally does
  3. Header order — real browsers send headers in a specific order that requests doesn’t replicate by default
  4. TLS fingerprint (JA3/JA4) — the cipher suite and extension order your TLS client sends. requests looks nothing like Chrome at this layer.
  5. HTTP/2 fingerprint — frame order, window sizes, headers
  6. Behavioral signals — mouse movement, scroll, timing

You can fix 1 and 2 with plain requests. For 3–5, you need a library that mimics a browser’s network stack:

# curl_cffi impersonates Chrome's TLS and HTTP/2 fingerprint
from curl_cffi import requests

resp = requests.get(url, impersonate="chrome124")

For level 6, you need an actual browser — Playwright or Selenium — often with a stealth plugin that patches navigator.webdriver and similar tells.

Rule of thumb: start minimal. If curl_cffi with impersonate="chrome124" and a sane Accept-Language gets past a site, don’t over-engineer. Real browsers are expensive; only pay that cost when you have to.