What you need
curl and coreutils, present on any Linux server and on macOS with Homebrew. curl speaks HTTP and SOCKS5 proxies with one flag, -x. For a package, allow the address of that machine in the ProxyMix panel first; the proxies need no logins after that.
One request through a random proxy
The first script downloads the list, takes a random line with shuf and sends one request through it. tr removes Windows line endings if they appear.
LIST_URL="YOUR_LIST_LINK"
PROXY=$(curl -fsS "$LIST_URL" | tr -d '\r' | shuf -n 1)
curl -x "socks5h://$PROXY" https://ifconfig.me/ipA loop over many URLs
The second script reads urls.txt and sends each URL through its own random proxy. It prints the HTTP code, the URL and the proxy, so failures show up at once.
LIST_URL="YOUR_LIST_LINK"
curl -fsS "$LIST_URL" | tr -d '\r' > proxies.txt
while read -r url; do
proxy=$(shuf -n 1 proxies.txt)
curl -s -m 20 -x "socks5h://$proxy" -o /dev/null -w "%{http_code} $url via $proxy\n" "$url"
done < urls.txtA list file kept fresh by cron
For programs that read a file, this line downloads the list every 10 minutes and replaces the file only after a complete download.
*/10 * * * * curl -fsS "YOUR_LIST_LINK" -o /opt/proxies/list.tmp && mv /opt/proxies/list.tmp /opt/proxies/proxies.txtResidential lines with a login
curl takes the login of a residential line with -U, so the password needs no encoding even with : or @ in it. The script splits the line, sends three requests and prints the address each one came from.
LINE="HOST:PORT:LOGIN:PASSWORD"
SCHEME="http"
IFS=: read -r GATE GATE_PORT USER_ID USER_KEY <<< "$LINE"
for i in 1 2 3; do
curl -sS -m 20 -x "$SCHEME://$GATE:$GATE_PORT" -U "$USER_ID:$USER_KEY" https://ifconfig.me/ip
echo
doneEvery curl call opens a new connection, so a rotating line shows a new IP on each request and a sticky line shows the same one. For a SOCKS5 line set SCHEME="socks5h": -U works for SOCKS5 logins too.
Parallel requests
The loop sends one request at a time. To go faster, feed the URLs to xargs -P or GNU parallel and keep the number of processes within the package threads. Switch socks5h:// to http:// in the -x flag if a target works better over HTTP.
Reading curl errors
- Exit code 7: curl could not reach the proxy. Make sure the panel lists the address of this machine.
- Exit code 56 with "CONNECT tunnel failed, response 407": the login or the password of a residential line is wrong.
- Exit code 28: the request timed out. The line is slow, so retry through another one or raise
-m. - Exit code 97: the SOCKS handshake failed. Retry through another line or switch to
http://. - HTTP 403 or 429 from the target: the site limits the address, so spread requests over more lines.
Next steps
For a full scraper see the Python guide or the Node.js guide. Programs such as ScrapeBox and Key Collector read the same file that cron keeps fresh.