If you are monitoring a large number of sites on the same IP address, you can configure different crawl times for each site.
For SITEMAP, HTTP, MONITORING and WEBPERF, you can choose the start time for the crawl.
- You can configure the crawl of one site every hour, i.e. one site at midnight, one site at 1am, one site at 2am, and so on.
- You can run the tools on different days. One tool on Monday, another on Tuesday, and so on.
Our SENTINEL crawler is deliberately slow by default. One URL per second, just one thread, and one site after another.
ALERTING crawls every hour; it is recommended to include only a few pages per template.
However, both ALERTING and MONITORING crawl a site’s URLs one after the other using just one thread. This remains fairly gentle…
If you have a lot of sites, you should avoid putting too many URLs on ALERTING and customise the start times for MONITORING so that nothing overlaps.
With all these tips, your server should cope very easily.
By way of comparison, remember that the most popular crawler on the market crawls 2 URLs per second and runs 5 threads!
