A spider is a program used by a search engines to index the World Wide Web content. Spiders captures specific information about a page, some capture the title and the first 1,000 characters of content, some captures the title and "description" meta tag, some look for only the keywords and some use the combination of all of these methods.