proxymix
Code examples

Proxy rotation in Python: urllib, requests and SOCKS5

Two working examples: one uses only the standard library, the other uses requests with SOCKS5. Both take a random line from your list for each request.

What you need

Python 3 and a ProxyMix list link for a package, or a residential line from the panel. For a package, allow the address of the machine with the script in the panel; the proxies recognise it, so that code passes no logins. The first example uses only the standard library and runs on any Python 3. The second uses requests with SOCKS5 support.

Install the packages for the second example: pip install "requests[socks]".

Standard library and an HTTP proxy

The script downloads the list once, picks a random line and sends one request through it. The reply from ifconfig.me shows the address that sites see.

Standard library, HTTP
import random
from urllib.request import ProxyHandler, build_opener, urlopen

LIST_URL = "YOUR_LIST_LINK"

lines = urlopen(LIST_URL, timeout=30).read().decode().split()
picked = random.choice(lines)
endpoint = f"http://{picked}"
opener = build_opener(ProxyHandler(dict.fromkeys(("http", "https"), endpoint)))
seen_as = opener.open("https://ifconfig.me/ip", timeout=20).read().decode()
print(f"{picked} -> {seen_as}")

requests with SOCKS5

The second example sends three requests, each through its own random proxy. With the socks5h scheme the proxy resolves domain names, and your server sends no DNS query to the target.

requests with SOCKS5
import random
import requests

LIST_URL = "YOUR_LIST_LINK"

pool = requests.get(LIST_URL, timeout=30).text.split()
with requests.Session() as session:
    for line in random.sample(pool, k=min(3, len(pool))):
        session.proxies = dict.fromkeys(("http", "https"), f"socks5h://{line}")
        answer = session.get("https://ifconfig.me/ip", timeout=20)
        print(f"{line} -> {answer.text.strip()}")

Residential lines with a login

A residential line carries its own login, so the code passes it in the proxy URL. quote encodes characters such as @ and : in the login and password, otherwise requests would split the URL in the wrong place. Paste a line from the panel into LINE.

requests with a residential line
import requests
from urllib.parse import quote

LINE = "HOST:PORT:LOGIN:PASSWORD"

host, port, login, password = LINE.split(":", 3)
proxy = f"http://{quote(login, safe='')}:{quote(password, safe='')}@{host}:{port}"
for _ in range(3):
    answer = requests.get("https://ifconfig.me/ip", proxies={"http": proxy, "https": proxy}, timeout=20)
    print(answer.text.strip())

Each requests.get opens a new connection, so a rotating line answers with a new address every time. Inside a Session requests keep the connection alive, and they leave through one IP until it closes. A sticky line keeps its address either way. For a SOCKS5 line, change the scheme to socks5h://.

From an example to a scraper

  • In long runs fetch the list again every 8 to 10 minutes. Hold it in a variable and replace it on a timer rather than downloading it before every request.
  • Catch timeouts and connection errors and retry through another line, since some lines of a big list will be slow.
  • Keep concurrency below the package threads. With a thread pool or asyncio, set the pool size under the limit.
  • For asyncio take aiohttp with an HTTP proxy or a SOCKS connector package. The rotation logic stays the same.

Errors you may see

  • ProxyError or a refused connection on every line: the panel does not have the address of your machine yet.
  • ProxyError with "Tunnel connection failed: 407": the login or the password of a residential line is wrong.
  • ConnectTimeout or ReadTimeout: the line is slow or has left the pool, retry through another one.
  • A 403 or a captcha page: the target limits the address, so spread the load over more lines and slow down.
  • SSLError through a proxy: usually a broken line, retry through another one.

Next steps

The Node.js guide and the curl guide show the same rotation in other languages. Packages and threads are on the prices page, and the free proxy list helps to test the code before you buy.

Questions

Questions

All questions
How often should the script reload the list?

Once per 8 to 10 minutes for long runs. A short script can load it once at the start.

What about asyncio?

Use the same approach with aiohttp: hold the lines in a variable and give every request its own proxy.

Which proxy scheme should I use in requests?

socks5h when the proxy should resolve domain names, http:// for plain HTTP proxies. Both work with every package line.

How do I pass a residential login in requests?

Put it into the proxy URL as http://login:password@host:port and encode the login and the password with urllib.parse.quote.

Ready to try it?

Top up your balance, create a list with the type and country you need, and test the pool on your site. A trial gigabyte costs from $0.27 and traffic lasts three months.

Build an order