When Netpeak tools need proxies
Netpeak Spider crawls sites, and Netpeak Checker pulls search results and metrics for lists of URLs and keywords. An audit of your own site rarely needs proxies. Crawls of large competitor sites and mass search checks do: without proxies the target or the search engine limits your IP, and the run slows down or stops.
Both tools take a proxy list in their settings and rotate it between requests. For a package, allow your computer address in the ProxyMix panel first.
Step by step
- Download the list into a file or copy it from the link.
- Open the proxy settings in Spider or Checker and load the list from the file or the clipboard.
- Use the lines as HTTP proxies: Spider takes a list of HTTP proxies, and every package port answers HTTP.
- Set the threads within the package limit.
- Run a short test crawl and look at the share of errors.
Spider and Checker settings
Spider sends requests to one site, so the crawl speed depends on how many requests that site tolerates per address. Raise the threads step by step and watch for 429 and 403 answers. Checker spreads requests across search engines and services and gains the most from a long list.
Requests through a proxy take longer than direct ones. Raise the timeout a little and add threads to keep the pace.
A file that stays fresh
Yesterday’s copy of the list holds many dead lines. Reload it before each big run, or keep a file that a scheduler rewrites every 10 minutes:
*/10 * * * * curl -fsS "YOUR_LIST_LINK" -o /opt/proxies/list.tmp && mv /opt/proxies/list.tmp /opt/proxies/proxies.txtTips
- Use the same file for Spider and Checker.
- Pick World MIX for global search checks and the Europe or CIS pool for local results.
- Turn proxies off for audits of your own site to keep the crawl simple.
Typical jobs
- Crawling a large competitor site without hitting its per-IP limit.
- Checking positions and snippets for thousands of keywords in Checker.
- Pulling metrics for URL lists from search engines and services.
- Checking how a site answers visitors from different countries.
Residential lines in Netpeak
Spider takes a list of host:port:login:password lines, the same form the panel gives, but the host has to be an IPv4 address. Run nslookup on the host of a residential line and put the IP in its place. Use HTTP lines.
A “Country” line with a chosen country shows how a site answers visitors from that country, and Checker can pull search results through home addresses when an engine blocks data center ranges. Crawls download whole pages, so estimate the gigabytes before a big run.
Test first
Once your account is verified, ask support for a free hour and crawl a real target. Compare packages on the prices page, and for custom crawlers see the Python guide.