### robots.txt for http://www2.math.duke.edu etc User-agent: * Crawl-delay: 5 Disallow: /cgi-bin Disallow: /perl-bin Disallow: /jam Disallow: /acjobs Disallow: /mpo #Disallow: /photos Disallow: /archive Disallow: /tw Disallow: /sqltut.htm Disallow: /filename.html Disallow: /manual #Disallow: /db/aas/cs #Block bots User-agent: The Knowledge AI Disallow:/