The Beta News

If you run a blog or a website you probably know what a robots.txt file is, so you can skip the next line. It’s a small text file that tells search engine what to include and not to include in their indexing of your site.

As you might have noticed or read www.whitehouse.gov has been given an Obama make-over. The changes has aren’t just graphical and textual. Perhaps as a comparison to the previous government, the new robots.txt is a bit more welcoming.

This is Obama’s robots.txt:

User-agent: *
Disallow: /includes/

Yes that’s it!

Now compare it to George W. Bush’s 2400 line robots.txt (I won’t post it all)

1. User-agent:     *
2. Disallow:       /cgi-bin
3. Disallow:       /search
4. Disallow:       /query.html
5. Disallow:       /omb/search
6. Disallow:       /omb/query.html
7. Disallow:       /expectmore/search
8. Disallow:       /expectmore/query.html
9. Disallow:       /results/search
10. Disallow:       /results/query.html
11. Disallow:       /earmarks/search
12. Disallow:       /earmarks/query.html
13. Disallow:       /help
14. Disallow:       /360pics/text
15. Disallow:       /911/911day/text
16. Disallow:       /911/heroes/text
17. Disallow:       /911/messages/text
18. Disallow:       /911/patriotism/text
19. Disallow:       /911/patriotism2/text

Let’s hope Obama keeps it this way. It’s a good start. So American…please don’t screw this up