proxymix
Software setup

Proxies for Netpeak Spider and Checker: list setup

Netpeak tools take a list of proxies in the settings and rotate them between requests. That helps on large audits and SERP checks. Residential lines need an IPv4 host and a login.

When Netpeak tools need proxies

Netpeak Spider crawls sites, and Netpeak Checker pulls search results and metrics for lists of URLs and keywords. An audit of your own site rarely needs proxies. Crawls of large competitor sites and mass search checks do: without proxies the target or the search engine limits your IP, and the run slows down or stops.

Both tools take a proxy list in their settings and rotate it between requests. For a package, allow your computer address in the ProxyMix panel first.

Step by step

  1. Download the list into a file or copy it from the link.
  2. Open the proxy settings in Spider or Checker and load the list from the file or the clipboard.
  3. Use the lines as HTTP proxies: Spider takes a list of HTTP proxies, and every package port answers HTTP.
  4. Set the threads within the package limit.
  5. Run a short test crawl and look at the share of errors.

Spider and Checker settings

Spider sends requests to one site, so the crawl speed depends on how many requests that site tolerates per address. Raise the threads step by step and watch for 429 and 403 answers. Checker spreads requests across search engines and services and gains the most from a long list.

Requests through a proxy take longer than direct ones. Raise the timeout a little and add threads to keep the pace.

A file that stays fresh

Yesterday’s copy of the list holds many dead lines. Reload it before each big run, or keep a file that a scheduler rewrites every 10 minutes:

crontab
*/10 * * * * curl -fsS "YOUR_LIST_LINK" -o /opt/proxies/list.tmp && mv /opt/proxies/list.tmp /opt/proxies/proxies.txt

Tips

  • Use the same file for Spider and Checker.
  • Pick World MIX for global search checks and the Europe or CIS pool for local results.
  • Turn proxies off for audits of your own site to keep the crawl simple.

Typical jobs

  • Crawling a large competitor site without hitting its per-IP limit.
  • Checking positions and snippets for thousands of keywords in Checker.
  • Pulling metrics for URL lists from search engines and services.
  • Checking how a site answers visitors from different countries.

Residential lines in Netpeak

Spider takes a list of host:port:login:password lines, the same form the panel gives, but the host has to be an IPv4 address. Run nslookup on the host of a residential line and put the IP in its place. Use HTTP lines.

A “Country” line with a chosen country shows how a site answers visitors from that country, and Checker can pull search results through home addresses when an engine blocks data center ranges. Crawls download whole pages, so estimate the gigabytes before a big run.

Test first

Once your account is verified, ask support for a free hour and crawl a real target. Compare packages on the prices page, and for custom crawlers see the Python guide.

Questions

Questions

All questions
Do Spider and Checker use the same list?

Yes, load the same file into both.

Why are some checks slower with proxies?

Every request goes through another server. Raise the timeout a little and add threads.

Does Netpeak work with SOCKS5?

Netpeak Spider takes a list of HTTP proxies. Every package port answers HTTP, and residential lines should be created as HTTP.

Why does Netpeak reject a residential line?

The host field takes only an IPv4 address. Replace the host name of the line with its IP from nslookup.

Ready to try it?

Top up your balance, create a list with the type and country you need, and test the pool on your site. A trial gigabyte costs from $0.27 and traffic lasts three months.

Build an order