LWP::RobotUA

LIB::LWP::ROBOTUA(1)  User Contributed Perl Documentation LIB::LWP::ROBOTUA(1)



NAME
       LWP::RobotUA - A class for Web Robots

SYNOPSIS
         require LWP::RobotUA;
         $ua = new LWP::RobotUA 'my-robot/0.1', 'me@foo.com';
         $ua->delay(10);  # be very nice, go slowly
         ...
         # just use it just like a normal LWP::UserAgent
         $res = $ua->request($req);


DESCRIPTION
       This class implements a user agent that is suitable for robot
       applications.  Robots should be nice to the servers they visit.  They
       should consult the robots.txt file to ensure that they are welcomed and
       they should not send too frequent requests.

       But, before you consider writing a robot take a look at
       <URL:http://info.webcrawler.com/mak/projects/robots/robots.html>.

       When you use a LWP::RobotUA as your user agent, then you do not really
       have to think about these things yourself.  Just send requests as you
       do when you are using a normal LWP::UserAgent and this special agent
       will make sure you are nice.

METHODS
       The LWP::RobotUA is a sub-class of LWP::UserAgent and implements the
       same methods.  The use_alarm() method also desides whether we will wait
       if a request is tried too early (if true), or will return an error
       response (if false).

       In addition these methods are provided:

       $ua = LWP::RobotUA->new($agent_name, $from, [$rules])

       Your robot's name and the mail address of the human responsible for the
       robot (i.e. you) is required by the constructor.

       Optionally it allows you to specify the WWW::RobotRules object to use.

       $ua->delay([$minutes])

       Set the minimum delay between requests to the same server.  The default
       is 1 minute.

       $ua->rules([$rules])

       Set/get which WWW::RobotRules object to use.

       $ua->no_visits($netloc)

       Returns the number of documents fetched from this server host.

       $ua->host_wait($netloc)

       Returns the number of seconds you must wait before you can make a new
       request to this host.

       $ua->as_string

       Returns a text that describe the state of the UA.  Mainly useful for
       debugging.

SEE ALSO
       the LWP::UserAgent manpage, the WWW::RobotRules manpage

AUTHOR
       Gisle Aas <aas@sn.no>




3rd Berkeley Distribution    perl 5.003, patch 07         LIB::LWP::ROBOTUA(1)