In order to avoid undesirable articles in the research indexes, webmasters can instruct spiders to not crawl certain files or directories throughout the regular robots.txt file in the foundation directory of the domain. On top of that, a web site could be explicitly excluded from a online search engine's database https://news.rafeeg.ae