In order to avoid undesirable content while in the research indexes, website owners can instruct spiders to not crawl certain files or directories in the standard robots.txt file in the root Listing from the domain. On top of that, a site could be explicitly excluded from a search engine's databases by using a meta tag distinct to robots (normally