Introduction:
======
This is the robot we use to compile the contributed blacklists @
*tp://ftp.ost.eltele.no/pub/www/proxy/squidGuard/contrib/blacklists.tar.gz

Sorry, at present this is the only documentation for the
squidGuardRobot.  Maybe I some day find time to do the doc job. In the
meantime here is a little introduction that may clear out things:

As of squidGuardRobot-2.2.2 the new robot seems functioning properly
and stable.

How it works:
=============
 * First it opens all the private .db files.
 * Loads all new entries from the input files.
 * Removes all new exception matches.
 * Removes all expired, redundant entries.
 * Downloads all sources that haven't been downloaded for a while.
 * Parses and extracts all links and stores new ones to the link.db
   file.
 * These two steps are done in bulks of 512 (configurable) to avoid
   being to hard on RAM.
 * Then tests all links that haven't been tested for a while with a
   HEAD request.
 * This is done in bulks of 512 (configurable) to avoid being to hard
   on RAM.
 * Then compiles all new and refreshed links into domain and urllists
 * Then removes all latent redundancy.
 * And finally dumps the keys from the domain and urllists to plain
   text files
 * In all steps lots of status info is recorded in the value fields of
   the .db files.

Configuration/maintenance:
==========================
 * Put web links (http://...) in the file "link" (optional)
 * Put linkpage URLs (http://sexlinks...) in the file "source"
   (optional)
 * Put whole domains that should be skiped (recursively) (cnn.com)
   into the file "domainexception" (optional)
 * Put (squidguard style) urls that should be skiped (some.site/good)
   into the file "urlexception" (optional)
 * Put exact match (squidguard style) urls and/or domains that should
   be skiped (some.site/this) into the file "exception" (optional)
 * Put (squidguard style) urls that should be included anyway
   (dir.yahoo.com/Business_and_Economy/Companies/Sex) in the file
   "include" (optional)
 * All these files are loaded into ditto .db files and truncated at
   startup.
 * Put all dirty words etc. you can imagine as perl regex (similar to
   the squidguard expressionlist but probably with a broader scope)
   into the file "patterns". The expressions are used to chose the
   best/broadest possible entry from a single link. (optional)

Well, if you don't use any of these options, then nothing very
exciting will happen at all :-o

The files domain.db and url.db are internal and _not_ directly
compatible with squidguard.

The output files are:
 * candidate - when a source has bounced with redirect the new url
   candidate is shown here
 * domains - the squidguard domainlist
 * urls - the squidguard urllist

Starting:
=========
To avoid changing the paths, timeouts etc. over and over for each
robot release you may put your own settings into a config file and run
(nice is nice unless you have a dedicated server):
   nice squidGuardRobot -c config >log 2>&1 &

In v2 I dropped most of the v1 command line options since I ended up
with more options than /bin/ls ;-) Note: The config file is simply
straight Perl code that is eval'ed line by line to overrule the
default settings, so if you change the value for $home then you'll
have to explicitly redefine all the paths that depends on $home as
well!

In the beginning you probably want to run in verbose or debug mode to
see and learn whats going on:
   nice squidGuardRobot -d -c config >log 2>&1 & tail -f log

Be prepared to have it runing for _several_ hours. Especially the
first times when all links are unknown and have to be tested.

Should I use it?
================
In contrast to squidguard, the squidGuardRobot will eat your CPU, RAM
and bandwidth for breakfast, lunch and supper as well;-) Though there
may be reasons why you want to run it locally anyway, like:
 - You are well connected to the net.
 - You have a powerful spare server.
 - The contributed blacklists doesn't match your needs at all.
 - You want to make "whitelists" like financial etc.
 - You are a masochist

For the average site though it is normally better to contribute to the
common blacklists.  Any contributions to the robot config and
corrections to the blacklists (both links, sources, exceptions and new
categories) are always welcome. See:
   http://www.squidguard.org/faq/#blacklist_project
   http://www.squidguard.org/faq/#blacklist_contrib
   http://www.squidguard.org/blacklist/

Pl
