email when search engines spider

Private Joined: Mar 10, 2003 Posts: 40

Insert in a .php page that will email you when Google and other search engines are indexing that page
On my page I added this to the index.php file but it could be added anywhere you want.

Code:

bot.php (change the $to var with your email address)
Code / Sample:
<?php

$botlist = array(
"Teoma",
"alexa",
"froogle",
"inktomi",
"looksmart",
"URL_Spider_SQL",
"Firefly",
"NationalDirectory",
"Ask Jeeves",
"TECNOSEEK",
"InfoSeek",
"WebFindBot",
"girafabot",
"crawler",
"www.galaxy.com",
"Googlebot",
"Scooter",
"Slurp",
"appie",
"FAST",
"WebBug",
"Spade",
"ZyBorg",
"rabaz");

foreach($botlist as $bot) {

if(ereg($bot, $HTTP_USER_AGENT)) {

if($bot == "Googlebot") {
if (substr($REMOTE_HOST, 0, 11) == "216.239.46.") $bot = "Googlebot Deep Crawl";
elseif (substr($REMOTE_HOST, 0,7) == "64.68.8") $bot = "Google Freshbot";
}
if ($QUERY_STRING != "") {
$url = "http://" . $SERVER_NAME . $PHP_SELF . "?" . $QUERY_STRING . "";
} else {
$url = "http://" . $SERVER_NAME . $PHP_SELF . "";
}

// settings
$to = "email@your-domain.com";
$subject = "Detected: $bot on $url";
$body = "$bot was deteched on $url\n\n
Date.............: " . date("F j, Y, g:i a") . "
Page.............: " . $url . "
Robot Name.......: " . $HTTP_USER_AGENT . "
Robot Address....: " . $REMOTE_ADDR . "
Robot Host.......: " . $REMOTE_HOST . "
";

mail($to, $subject, $body);

}

}

?>

“I did not code this just sharing it”

Private Joined: Mar 10, 2003 Posts: 40

Another good tool that I recently installed to watch Search engine spiders

Quote:

GoogleStats is a free Open Source PHP application analyzing the visits of Googlebot any many more robots on your site. The idea is quite simple: you just have to include a PHP script on each page of your site (you can choose the pages where you include it, only those ones will be analyzed). When Googlebot (or any other robot) comes to crawl one page, its visit is recorded in a MySQL database.

The second part of the application displays the statistics for a given period of time. Use the calendar on the left to select a day, a week or even a month. Three types of analysis are given:

Results:
nb of visits of each robot
nb of different pages viewed by each robot
average nb of visits per day
list of all the robots detected during the defined period
list of the IP addresses of each detected robot
Pages:
list of all pages viewed by each robot within the analyzed period, with URL, date, number of visits, IP address and DNS. You may sort these columns by clicking on its header.
Graph: displays in a bar diagram the statistics of the selected robot's visits.

http://www.googlestats.com/english/index.htm

Lieutenant Joined: Apr 09, 2003 Posts: 188

Antipas can u help me? i tryed to insert the google stats but the code when i put in my nuke theme.php crashes my rigth blocks .

im using phpnuke 6.0

Private Joined: Mar 10, 2003 Posts: 40

Quote:

Uncompress the file 'googlestats.zip'. A directory named 'googlestats'
should be created. Once all the stuff listed above is done, transfer it via
FTP to the root of your web site.

Edit the file 'googlestats/admin/config.php' with a simple text editor
like Notepad and replace the 4 following elements by the real values
(if you need help you may ask your host provider):
- "HOST" : the MySQL hostname
- "LOGIN" : your login
- "PASSWORD" : your password (to access the database)
- "BASE" : the MySQL database name

Fill in your email address (this address will be used to send you the Full
Crawl alerts or to warn you about MySQL execution errors) : variable
ADRESSE_EMAIL

The options of GoogleStats are configured by the following variables.
If you want, you may modify some of them:
- URL_REWRITING : put "y" if you're using it, otherwise "n"
- GRAPH_SCALE : put "lin" (linear) or "log" (logarithmic) for the
scale of the graphs
- VALEURS_GRAPH : put "y" if you want to display the values on the
graph. Otherwise put "n"
- SET_EXEC_TIME : put "y" if you want to set a maximum execution time
for the graph script. Otherwise put "n"
- TEST_FULL_CRAWL : put "y" if you want to detect when the Full Crawl
begins and to be alerted by email (the email will be
sent to the address ADRESSE_EMAIL). Otherwise put "n"
- SEND_ERROR_MYSQL : put "y" to receive emails when an MySQL error occurs.
Otherwise put "n".

To customize the installation, you can also modify these variables:
- LANGUE : put "en" (english), "fr" (french) or "de" (deutsch)
- TABLE_LOG : name of the MySQL table used to store the visits
- TABLE_ROBOTS : name of the MySQL table used to store the robots
- NB_J_DET_FULL_CRAWL : for the Full Crawl detection: an alert is sent to
warn the webmaster that the Full Crawl has begun as
soon as a Full Crawl robot's visit is detected and no
visit of the Full Crawl robot has been recorded in
the last NB_J_DET_FULL_CRAWL days.

4- Transfer the whole directory 'googlestats' by FTP to your server.

5- Protect the directory 'admin' (that you should have renamed) with a file
'.htaccess' and a file '.htpassword'.

6- Installation:
* If this is the first time that you install GoogleStats, go to the URL
http://__your_site__/googlestats/install.php and follow the instructions.
Once the installation process is over, delete the file 'install.php' from
your server.

7- Now you must insert the following code in each page of your site (apart from
the pages you don't want to be analyzed). If you're using Dreamweaver (or
any other software supporting templates), you can insert this code in your
template. In this case, use the code display and insert this code in a
non-editable zone:

<?php
include($DOCUMENT_ROOT.'/googlestats/admin/config.php');
include($DOCUMENT_ROOT.'/googlestats/googlestats.inc.php');
?>

I added it to the bottom of my index.php

Code:

include("footer.php");
}
}
include($DOCUMENT_ROOT.'/googlestats/admin/config.php');
include($DOCUMENT_ROOT.'/googlestats/googlestats.inc.php');

?>