To prevent undesirable material within the search indexes, webmasters can instruct spiders to not crawl certain information or directories with the regular robots.txt file in the basis Listing with the area. Also, a web site may be explicitly excluded from a online search engine's database by making use of a https://news.rafeeg.ae