To stop undesirable content during the look for indexes, site owners can instruct spiders not to crawl specific documents or directories with the standard robots.txt file in the foundation Listing with the area. Also, a site is usually explicitly excluded from the internet search engine's database through the use of a meta tag particular to robots