proxymix
Code examples

curl and bash with a proxy list: requests, loops, cron

curl speaks HTTP and SOCKS5 proxies with one flag. These scripts take random lines from your list, for a quick test or for a loop over many URLs.

What you need

curl and coreutils, present on any Linux server and on macOS with Homebrew. curl speaks HTTP and SOCKS5 proxies with one flag, -x. For a package, allow the address of that machine in the ProxyMix panel first; the proxies need no logins after that.

One request through a random proxy

The first script downloads the list, takes a random line with shuf and sends one request through it. tr removes Windows line endings if they appear.

One request through a random proxy
LIST_URL="YOUR_LIST_LINK"
PROXY=$(curl -fsS "$LIST_URL" | tr -d '\r' | shuf -n 1)
curl -x "socks5h://$PROXY" https://ifconfig.me/ip

A loop over many URLs

The second script reads urls.txt and sends each URL through its own random proxy. It prints the HTTP code, the URL and the proxy, so failures show up at once.

Loop over urls.txt
LIST_URL="YOUR_LIST_LINK"
curl -fsS "$LIST_URL" | tr -d '\r' > proxies.txt
while read -r url; do
  proxy=$(shuf -n 1 proxies.txt)
  curl -s -m 20 -x "socks5h://$proxy" -o /dev/null -w "%{http_code} $url via $proxy\n" "$url"
done < urls.txt

A list file kept fresh by cron

For programs that read a file, this line downloads the list every 10 minutes and replaces the file only after a complete download.

crontab
*/10 * * * * curl -fsS "YOUR_LIST_LINK" -o /opt/proxies/list.tmp && mv /opt/proxies/list.tmp /opt/proxies/proxies.txt

Residential lines with a login

curl takes the login of a residential line with -U, so the password needs no encoding even with : or @ in it. The script splits the line, sends three requests and prints the address each one came from.

curl with a residential line
LINE="HOST:PORT:LOGIN:PASSWORD"
SCHEME="http"
IFS=: read -r GATE GATE_PORT USER_ID USER_KEY <<< "$LINE"
for i in 1 2 3; do
  curl -sS -m 20 -x "$SCHEME://$GATE:$GATE_PORT" -U "$USER_ID:$USER_KEY" https://ifconfig.me/ip
  echo
done

Every curl call opens a new connection, so a rotating line shows a new IP on each request and a sticky line shows the same one. For a SOCKS5 line set SCHEME="socks5h": -U works for SOCKS5 logins too.

Parallel requests

The loop sends one request at a time. To go faster, feed the URLs to xargs -P or GNU parallel and keep the number of processes within the package threads. Switch socks5h:// to http:// in the -x flag if a target works better over HTTP.

Reading curl errors

  • Exit code 7: curl could not reach the proxy. Make sure the panel lists the address of this machine.
  • Exit code 56 with "CONNECT tunnel failed, response 407": the login or the password of a residential line is wrong.
  • Exit code 28: the request timed out. The line is slow, so retry through another one or raise -m.
  • Exit code 97: the SOCKS handshake failed. Retry through another line or switch to http://.
  • HTTP 403 or 429 from the target: the site limits the address, so spread requests over more lines.

Next steps

For a full scraper see the Python guide or the Node.js guide. Programs such as ScrapeBox and Key Collector read the same file that cron keeps fresh.

Questions

Questions

All questions
How do I use HTTP instead of SOCKS5?

Replace socks5h:// with http:// in the -x flag.

How do I run many requests at once?

Feed the URLs to xargs -P or GNU parallel and keep the number of processes within the package threads.

How do I pass a residential login to curl?

With -U login:password next to -x http://host:port, or socks5h://host:port for a SOCKS5 line.

Ready to try it?

Top up your balance, create a list with the type and country you need, and test the pool on your site. A trial gigabyte costs from $0.27 and traffic lasts three months.

Build an order