Introduction

Bots are also known as crawlers and spiders scour the web to find and index content. They keep search engines stocked with fresh webpages.