About the Dataclip Crawler
Why is Dataclip Crawling My Site?
Dataclip aims to crawl the top 10 million business websites on a monthly basis to keep up to date with the latest technology trends. By keeping a large, fresh index we are able to provide our customers with valuable aggregate information about technology trends as well as targeted lists of which sites are using what technologies. Read more »
How do I know Dataclip is Crawling My Site?
The Dataclip web crawler (sometimes also referred to as a "spider" or "robot") sends an identifying User-Agent header in each request it makes to your site. Most web servers log this information in their access logs. The Dataclip web crawler User-Agent header will look similar to the following:
Mozilla/5.0 (compatible; heritrix/3.0 +http://www.dataclip.com/crawler.html)
Can I keep Dataclip from Crawling My Site?
Yes. Send an email to info@dataclip.com and we can remove your site from our index and crawl list.
