a web crawler or spider is a computer program that automatically fetches the contents of a web page. the program then analyses the content, for example to index it by certain search terms. search engines commonly use web crawlers.