# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file # # To ban all spiders from the entire site uncomment the next two lines: Host: v-avto.ru User-Agent: * Allow: / Disallow: /shop/catalog Disallow: /admin Disallow: /catalog Disallow: /login Disallow: /reg Disallow: /restore Disallow: /basket Disallow: /orders Disallow: /user Disallow: /favorites Disallow: /prices Disallow: /search User-agent: Yandex Crawl-delay: 20 User-agent: AhrefsBot Disallow: /