Rotating proxies for web scraping: staying under rate limits

WinGate private IPv4 pool for scrapers and bots: unlimited proxies with HTTP, HTTPS and SOCKS5 in one access.

Rotating proxies for web scraping spread your requests across many IP addresses, so the site sees only a little traffic from each address. That matters because most rate limits are counted per IP. Send three hundred requests a minute from your office line and the site starts answering 429 Too Many Requests, then blocks the address. Send the same three hundred through a pool of a few thousand addresses and each one makes a request every few minutes, which looks like ordinary visitors.

If your scraper works on a hundred pages and stalls on ten thousand, the cause is almost always this counter.

The video shows the whole run: copy the pool link from the dashboard, put it into the script, set the thread count and start.

How sites count you: rate limits and IP bans

A rate limit is a counter. The server notes who sent a request and how many came from the same source in a time window. When the counter passes the limit, the server replies with HTTP 429. RFC 6585, the document that defines this code, says the response may carry a Retry-After header telling the client how long to wait. Well-behaved scrapers read that header and slow down.

The source a server counts is almost always the IP address. Cookies can be dropped and headers can change between requests, but every request arrives from an address. So the same scraper code behaves very differently depending on where its requests come from:

• one address: every request adds to one counter, and a few hundred pages trigger the limit;
• a rotating pool: the load is split across thousands of counters, and each stays far below the limit.

An address that keeps tripping the limit gets a longer ban, sometimes for hours. That is why scrapers that run from one server stall halfway through a crawl, while the same job through a pool finishes.

One address against a rotating pool

A single fixed address is fine for logging into one account or checking one page by hand. It is the wrong tool for collecting thousands of pages, because all the volume lands on one counter.

A rotating pool gives your software a list of addresses. Rotation can happen in two places. The provider can rotate behind one gateway, or your software can pick a new address from the list for every request. The second way gives you control: you decide when an address changes and you can keep one address for a whole login session when you need it.

HTTP, HTTPS or SOCKS5

Most scraping libraries speak HTTP proxies natively. Python requests, Scrapy and most desktop scrapers accept an address in the form http://IP:PORT and use the CONNECT method for HTTPS sites, so the page itself stays encrypted end to end.

SOCKS5 works one level lower and passes any TCP traffic, which is useful for software that is not a browser or an HTTP client: mail checkers, custom protocols, some antidetect tools. In requests you add SOCKS support with pip install requests[socks] and write socks5h:// so that DNS lookups also go through the proxy.

A pool that serves the same addresses over HTTP, HTTPS and SOCKS5 saves you from keeping separate lists for different tools.

Headers and user agents that match a real browser

Rotation hides the volume, but each request still has to look like it came from a normal browser. A few rules cover most cases:

• send the full User-Agent string of a current browser version, since sites recognise the library default at once;
• add Accept-Language and Accept headers that a browser with that User-Agent would send;
• keep the headers consistent for the life of one session, since a browser does not change its language between two clicks;
• respect robots.txt and the pacing a site asks for.

Changing the user agent on every request while the IP stays the same does more harm than good: the site sees one address pretending to be dozens of browsers.

The pool link in Python with threads

The script below loads the address list from the pool link once, then sends each URL through a random address from that list. It accepts both formats a dashboard can give: IP:PORT for access by IP binding and IP:PORT:LOGIN:PASS for access by login and password.

import random
import requests
from concurrent.futures import ThreadPoolExecutor

LIST_URL = "pool link from your dashboard"
THREADS = 100

proxies = requests.get(LIST_URL, timeout=15).text.split()
urls = open("urls.txt", encoding="utf-8").read().split()
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/141.0 Safari/537.36",
    "Accept-Language": "en-US,en;q=0.9",
}


def to_proxy(line):
    ip, port, *auth = line.split(":")
    if auth:
        return f"http://{auth[0]}:{auth[1]}@{ip}:{port}"
    return f"http://{ip}:{port}"


def fetch(url, tries=3):
    for _ in range(tries):
        p = to_proxy(random.choice(proxies))
        try:
            r = requests.get(url, headers=headers, proxies={"http": p, "https": p}, timeout=20)
        except requests.RequestException:
            continue
        if r.status_code == 429:
            continue
        return r.status_code, url
    return "failed", url


with ThreadPoolExecutor(max_workers=THREADS) as pool:
    for status, url in pool.map(fetch, urls):
        print(status, url)

A failed request or a 429 does not stop the job: the same URL is retried through another address. Raise THREADS step by step and keep it within the thread limit of your web scraping proxy plan.

Where a private pool fits

Public proxy lists look tempting for this, but their addresses are shared by everyone who found the same list, so many of them are already banned on popular sites before you send a single request. A private pool avoids that because its addresses do not appear in open lists.

WinGate is one such pool. It sells private IPv4 and SOCKS5 proxies that work over HTTP and HTTPS as well, with 12,000 to 15,000 addresses online on a typical day and a list that refreshes every 15 minutes. Plans allow up to 1,000, 3,000 or 5,000 threads, and traffic is not metered, so with unlimited proxies a long crawl does not run into a bandwidth bill. Access works by binding your own IP in the dashboard or with IP:PORT:LOGIN:PASS. You can check your own script against the pool with a free test of up to 2 hours before paying.

Settings that keep a long job stable

Concurrency

More threads finish faster only while the target site keeps up. Start at 50 to 100, watch the share of 429 and 5xx answers, and add threads while that share stays low. A thread count far above what the site can serve just turns into retries.

Session retention or instant rotation

Pick per task. Pages that need no login work best with a new address on every request. Anything with a login, a cart or a multi-step form needs one address for the whole session, so give each session its own address from the pool and keep it until the session ends.

Uptime and latency

Log three numbers per run: average response time, share of failed connections and share of 429 answers. If an address fails several times in a row, drop it from the in-memory list and reload the list from the pool link on the next run. A rising 429 share usually means the thread count is too high for that site, while timeouts point to the target's own load.

Before the first big run

Run the script on a few hundred URLs first and look at the log. If 429 answers stay rare and timeouts are few, raise THREADS and start the full list. Keep sessions with a login on one address and let everything else rotate per request.