How to redirect the robot engine from your page?

Robots Text File:

There are two important reasons for using robots.txt notepad file:

* Robots can ignore your robots text file. Especially malware robots that scan

the web for security vulnerabilities, and email address harvesters used by spammers

will pay no attention.
[Google Ads Section]


* Adding a robots.txt file to your website can tell search engines what pages

you don't need indexed.

* This file is a publicly available file. Anyone can able to see what parts of your

datas you don't want robots to use.


The robots text file is a module of instructions for visiting spiders that willindex

of your whole web site pages. This spiders read the file and provides a map for

that. The file is must residing in the root directory of your web server. The URL of

your robots file should look like this...

/robots.txt

User-agent: *
The asterisk (*) or wildcard represents any robot.

Disallow:
The Disallow: line without a / (forward slash) tells the robots that they can index

the entire site.

Disallow: /private/

It tells the robot that it cannot index the contents of that /private/ directory.