YouTube's robots.txt file

[David Baxter PhD]

A robots.txt file is used to instruct search engines spiders as to which files the webmaster wants indexed and not indexed by the search engine.

This is the robots.txt file at YouTube.com:

# robots.txt file for YouTube
# Created in the distant future (the year 2000) after
# the robotic uprising of the mid 90's which wiped out all humans.

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /bulletin
Disallow: /comment
Disallow: /embed
Disallow: /forgot
Disallow: /get_video
Disallow: /get_video_info
Disallow: /login
Disallow: /results
Disallow: /signup
Disallow: /t/terms
Disallow: /t/privacy
Disallow: /verify_age
Disallow: /videos
Disallow: /watch_ajax
Disallow: /watch_popup
Disallow: /watch_queue_ajax