Web Scraping Python

Proxies and Rotating IPs: When You Actually Need Them

· 1 min read

Most scraping tutorials reach for proxies on page one. In reality, you should reach for them last — after you’ve verified a single IP with a good User-Agent and sensible rate limit actually gets blocked.

When proxies are necessary:

  • You’re hitting a site that geo-fences by country
  • You’re running enough volume that one IP gets rate-limited regardless of pacing
  • The target specifically blocks cloud-provider IP ranges (common for sneaker, ticket, airline sites)

Using them in requests:

proxies = {
    "http":  "http://user:pass@proxy.example.com:8080",
    "https": "http://user:pass@proxy.example.com:8080",
}
session.get(url, proxies=proxies, timeout=15)

Rotation strategies, cheapest to most expensive:

  1. Datacenter proxies — pennies per GB, but frequently blocklisted
  2. ISP/static residential — look like home IPs but are stable; good middle ground
  3. Rotating residential — a fresh IP per request; most expensive, hardest to detect

One thing beginners miss: sticky sessions. If your workflow requires logging in or keeping a cart, you need the same IP across requests, not a random one per call. Most providers expose this as a session parameter in the proxy username (user-session-abc123).

And always test that your proxy actually rotates. Hit https://httpbin.org/ip from a few requests; if the IP doesn’t change, the provider is lying about the product.