How to Rotate Proxies in Python Requests
Automatically Rotate IPs

My Approach to Scraping and IP Rotation in Python

When you write scrapers or web monitors, you quickly learn that making too many requests from a single IP address is a bad idea. Target servers will rate-limit you, show you CAPTCHAs, or block your IP entirely. I have had my home IP blocked more times than I care to admit. The only reliable way to keep your scraper running is by rotating your proxies—switching the IP address for every request.

In this guide, I will show you how to build a simple, working proxy rotator in Python using the Requests library. We will fetch a live proxy list from our API, select proxies at random, and automatically handle failures so your script never gets stuck.

Step 1: Fetching Proxies Dynamically

Do not hardcode a list of proxies in your script. Free proxies go offline constantly, and hardcoded lists will stop working within a few hours. Instead, fetch them dynamically from a live source.

We provide raw text lists at Connect Proxy that update every hour. You can fetch HTTP proxies at https://connectoproxy.com/api/http.txt and SOCKS5 proxies at https://connectoproxy.com/api/socks5.txt. These URLs return a list of IP addresses separated by newlines, making them easy to parse in Python.

Here is how to fetch and parse this list into a Python list:

import requests
import random

def fetch_proxy_pool():
    api_url = "https://connectoproxy.com/api/http.txt"
    try:
        response = requests.get(api_url, timeout=8)
        if response.status_code == 200:
            # Split the plain text response by newlines
            raw_list = response.text.strip().split('\n')
            # Remove any extra whitespace and filter out empty lines
            proxy_list = [p.strip() for p in raw_list if p.strip()]
            print(f"Loaded {len(proxy_list)} proxies into your pool.")
            return proxy_list
    except Exception as e:
        print("Could not fetch the live proxy list:", e)
    return []

Step 2: Writing the Request Rotator

Now that we have a pool of proxies, we need a function that makes requests using them. Since free proxies can be unreliable, our rotator should pick a proxy at random, attempt the request, and check for errors. If a proxy fails or times out, the script should remove it from the pool and try another one.

This retry loop ensures that your script keeps running even if several proxies in a row are offline:

def request_with_rotation(url, proxy_pool):
    if not proxy_pool:
        print("No proxies available in your pool.")
        return None

    # Try up to 5 times with different proxies
    for attempt in range(5):
        proxy_ip = random.choice(proxy_pool)
        
        # Format the proxy dictionary for the Requests library
        proxies = {
            "http": f"http://{proxy_ip}",
            "https": f"http://{proxy_ip}"
        }

        print(f"Attempt {attempt+1}: Trying proxy {proxy_ip}...")
        
        try:
            # Set a low timeout so we do not wait too long for slow servers
            response = requests.get(url, proxies=proxies, timeout=4)
            if response.status_code == 200:
                print("Request succeeded!")
                return response
        except requests.exceptions.RequestException:
            print(f"Proxy {proxy_ip} failed. Removing from pool...")
            proxy_pool.remove(proxy_ip)

    print("All rotation attempts failed.")
    return None

Step 3: Putting It All Together

Here is how you can combine these two functions to scrape a website. We will send a request to https://httpbin.org/ip, which returns the origin IP address of the request. This lets you verify that the proxy rotation is actually working:

if __name__ == "__main__":
    target = "https://httpbin.org/ip"
    
    # 1. Get fresh proxies
    pool = fetch_proxy_pool()
    
    # 2. Make the request using rotation
    if pool:
        response = request_with_rotation(target, pool)
        if response:
            print("Response JSON:")
            print(response.json())

Developer Tips for Scraping

While the code above will get you started, here are a few things to keep in mind if you plan to scrape at scale:

• SOCKS5 Support: SOCKS5 is generally faster than HTTP for web scraping. To use SOCKS5 with the Requests library, you need to install the SOCKS dependency: pip install requests[socks]. Once installed, format your proxy dictionary as socks5://ip:port instead of http://.... For a detailed comparison of protocols, read our SOCKS Proxy Guide.
• Set Low Timeouts: Do not omit the `timeout` parameter in `requests.get()`. If you do, your script could hang indefinitely waiting for an offline proxy to respond. Keep timeouts between 3 to 5 seconds.
• Rotate User-Agents: Changing your IP address is only half the battle. If every request uses the same default Python user-agent header, websites will block you anyway. Rotate your user-agents along with your proxies.

For more details on request headers, timeouts, and sessions, check the official Python Requests Documentation.

If you need fresh proxies for your scrapers, bookmark the Connect Proxy Homepage and fetch our lists dynamically in your code.