Introduction

A text file inside the root directory that directs web robots (usually Yahoo crawlers) which web pages or portions of a website to examine and dismiss.