User-agent: * Crawl-delay: 1 #If proposed extended standard supported, at most 2 pages every second please Request-rate: 2/1