1 posts tagged "robots"

Google Newspaper 09 Sep 2008

Checking Google's robots.txt file, the file that tells other search engines what part of a website they can or cannot index, sometimes throws up interesting hints as to what might be coming next. Today a new entry has appeared under "newspaper". Guess it makes sense to want to start cataloguing the printed press...

Google, robots