diablo-files

DIABLO-FILES(5)               File Formats Manual              DIABLO-FILES(5)



NAME
       diablo-files - Synopsis of configuration and control files for Diablo

GENERAL
       This manual page provides a synopsis for various diablo configuration
       and control files.  It does not necessarily describe all commands or
       the specific file format.  You can obtain most of the latter by looking
       at the example config/control files in the sample/ directory.  What
       this manual page does is describe specific featurisms that would
       otherwise not be readily apparent.

       The location of all diablo files is controlled by /news/diablo.config.
       The location of diablo.config itself can be specified with the
       DIABLO_CONFIG_PATH environment variable or with the '-C
       diabloconfigpath' option to any command.  Diablo files are typically
       rooted at either the news root, the library root, or the database root
       (path_news, path_lib, or path_db in diablo.config).  See
       samples/diablo.config for more information.


DNEWSFEEDS
       The dnewsfeeds file specifies how diablo generates outbound feeds as
       well as specifies how diablo filters incoming feeds.  You specify each
       outbound feed with a label keyword, followed by various other keywords
       and ending with an end keyword.

       alias specifies an outbound newspath alias, which may contain '*' and
       '?' wildcards.  An inbound article whos path element matches any given
       alias for any given outbound feed is not requeued to that outbound
       feed.

       addgroup and delgroup filters articles to outbound feeds.  All
       add/delgroup commands are run against each newsgroup in the Newsgroups:
       header for the article and the last add/delgroup command effecting any
       given newsgroup is applied to the article.  Normally this means you
       start out more general at the beginning (i.e. '*') and become more
       specific at the end of your list.  When an article is cross posted to
       multiple newsgroups, success with any of the groups will cause the
       article to be propogated.

       NOTE!  If you 'addgroup *', then use delgroup and delgroupany to remove
       groups that you do not want, beware that control messages for ALL
       groups will still be propogated unless you also do a 'delgroup
       control.*'.

       delgroupany filters articles in a manner similar to delgroup, but if
       the specified groups exist in the Newsgroups: line *AT ALL*, no
       distribution to *any* of the other newsgroups will occur.  i.e. if you
       say 'delgroupany alt.warez*', it means that if an article is posted to
       comp.sys.amiga.misc and alt.warez*, the article will NOT be propogated.

       requiregroup An extremely exclusive filter that is the inverse of
       delgroupany.  The article's Newsgroups MUST contain a group matching
       the wildcard or it will not be propogated.  This is generally only used
       when splitting off control feeds.  Please see samples/dnewsfeeds for
       typical usage.

       groupref allows a feed to recursively include a group access list
       defined by a groupdef command.  The groupdef command may occur before
       OR after the newsfeed that references it, and groupdef's can be
       recursive.  see groupdef for more information.

       filter effects INBOUND feeds.  Articles coming FROM the specified label
       (see diablo.hosts on how to tie labels to incomming connections) are
       rejected if any listed newsgroup matches the wildcard specified in any
       of the filter commands.  You normally use this to reject articles cross
       posted to your local newsgroups that are propogated from outside
       entities.  Note that the history file is not updated, just in case the
       article is also brought in from a valid 'local' newsfeed.

       maxpath (1.08 or higher) effects OUTBOUND feeds.  Diablo will not send
       an article to an outbound feed if the article's Path: line contains
       more then the specified number of path elements.

       maxcross (1.04 or higher) effects OUTBOUND feeds.  Diablo will not send
       an article to an outbound feed if the article's Newsgroups: line
       contains more then the specified number of groups.

       maxsize (1.04 or higher) effects OUTBOUND feeds.  Diablo will not send
       an article to an outbound feed if the article is larger then the
       specified number of bytes.  You may use 'k', 'm', and 'g' to denote
       kilobytes, megabytes, and gigabytes.  For example, <I>maxsize 100k</I>
       .

       rtflush (1.12 or higher) effects OUTBOUND feeds.  Diablo will write the
       queue file for this feed unbuffered.  Generally used along with
       'realtime' in dnntpspool.ctl.

       There is an (unimposed) arbitrary limit of 256 add/delgroup entries per
       feed.  Currently this is due to the fact that Diablo scans the wildcard
       list linearly and cannot really support group-specific wildcards (where
       a feed wants 10,000 specific groups rather then fewer wildcarded
       groups).

       A special label, called ME may be specified.  This label generally only
       contains filter commands.  Diablo will revert to this label for any
       incoming connections that have not been associated with a specific
       label.

       There is also the groupdef command, which must occur outside any feed
       commands.  The groupdef command is followed by a set of groupadd,
       groupdel, groupdelany, and/or groupref commands to create a grouplist.
       The groupdef command is terminated by an end command.  Grouplist
       definitions may be referenced by feeds via the groupref command, and
       may be referenced by other grouplists.


DEXPIRE.CTL
       The dexpire.ctl file controls the length of time articles are stored
       for each group in the reader header database and the maximum number of
       articles that can be stored in the article index per group for
       dreaderd.  This index value is adjusted by dexpireover based on the
       number of articles in the newsgroup.

       The various options are separated by a colon and appear in any order
       after a wildmat specification of the newsgroup names. The options are:


       x to specify the number of days to keep articles matching this wildmat.
       Not specifying this option means that articles are not removed.
       Fractions of a day may also be used.


       a to override the default 512 for the starting 'maxarts' value per
       group matching this wildmat. The starting value is only used when the
       group index is first created.


       i to specify the minimum 'maxarts' that dexpireover may adjust to.


       j to specify the maximum 'maxarts' that dexpireover may adjust to.


DEXPIRE.FACTOR
       The dexpire.factor file is no longer used by Diablo and can be deleted.
       It may be used again in the future.


DNNTPSPOOL.CTL
       The dnntpspool.ctl file is used by the dspoolout program to manage the
       outbound queues.  Diablo will maintain a single outbound queue file for
       each feed.  dspoolout renames this file to a proper sequenced queue
       file, properly flushes Diablo's file descriptors, starts up dnewslink
       programs, and removes any sequenced queue files backlogged beyond the
       specified limit.

       This file allows you to specify the remote host, maximum number of
       queue files, and maximum number of dnewslink processes to run for each
       outbound feed, and other options.  Please refer to the
       samples/dnntpspool.ctl file.


DISTRIB.PATS
       The distrib.pats file is read and returned by the NNTP 'list
       distrib.pats' command.  It is also used to generate a Distribution:
       header internally when the POST command does not supply it (or supplies
       an empty Distributions header).


DISTRIBUTIONS
       The distributions file is read and returned by the NNTP 'list
       distributions' command.


DIABLO.HOSTS
       The diablo.hosts file controls authentication for incoming connections.
       Each line consists of a domain name or IP address wildcard and a label
       identifying which feed in dnewsfeeds the incoming connection is
       associated with.  THE LABEL IS NO LONGER OPTIONAL.  You must supply a
       label.

       Diablo normally requires that the reverse lookup match the forward
       lookup for security purposes, but many sites set up their reverse to
       point to a CNAME or set up their reverse to point to an unassociated
       host yet still request that you put a common hostname in your hosts
       file which 'resolves' to all the IP addresses of their news machines.

       Diablo will attempt to wildcard match the last two domain elements of
       the reverse domain name with non-wildcard domain names in diablo.hosts
       then issue a forward lookup of the name in diablo.hosts and attempt to
       match the IP.  In otherwords, if you have an entry for
       'newsfeeds.fubar.com' in diablo.hosts and an incoming connection's
       reverse lookup comes back 'news55.fubar.com', diablo will convert
       news55.fubar.com to *.fubar.com and attempt to match that against
       entries in diablo.hosts.  When a match occurs, it performs a forward
       lookup (in this case against 'newsfeeds.fubar.com') and tries to match
       up the IP address that way.

       This methodology has the advantage of not requiring diablo to do a
       sequential forward lookup of all the entries in diablo.hosts.  Each
       connection's DNS load is consistent only with the domain/IP the
       connection is coming from which is very important for stability in the
       face of a large number of feeds.


DREADER.HOSTS
       The dreaderd.hosts file contains access permissions for NNTP readers
       for the dreaderd server.  This file also contains access permissions
       for header-only feeds coming into the dreaderd server.


DSERVER.HOSTS
       The dserver.hosts file contains the outgoing spool and posting server
       configuration which dreaderd uses to make connections to outside
       servers for message retrieval and outgoing POSTed message propogation.


DQUEUE (directory)
       The dqueue directory contains the queue files, both the ones generated
       by the diablo server and the ones maintained by dspoolout.  Files in
       this directory are generally named as the label (diablo outbound queue
       file for a label), as the label.Snnnnn (sequenced queue file maintained
       by dspoolout), or .label.seq (sequence number information maintained by
       dspoolout).

       Note that the diablo queue format has changed as of V1.08.  Older
       versions of diablo dumped the filename and message id.  As of V1.08, an
       third field formatted as OFFSET,BYTES is added to support the new
       multi-article spool files.  DNewslink understands both formats.


FEEDS (directory)
       The feeds directory contains automatic group add/del information as
       requested by a feed through the feedrset and other Diablo commands.  If
       a file exists for any given feed, it overrides the addgroup and
       delgroup commands in the dnewsfeeds file for that feed.


NEWS SPOOL (/news/spool/news) (directory)
       Diablo implements a two-level news spool.  A directory of the form
       D.xxxxxxxx is created on the first level every 10 minutes.  Each
       discrete fork creates a distinct file in one or more of these
       directories when it receives an article.  The directory is chosen based
       on the expiration and the filename is chosen starting with a hash of
       the incoming IP.  The diablo process then exclusively locks the file(s)
       in question.  In the case of contention, the loser will generate
       another filename and loop until it finds or creates one.  The files or
       of the form B.xxxx where xxxx is basically random.  Multiple articles
       may be stored in each file.  An ascii NUL (code 0x00) is used as an
       out-of-band article separator.  The history and outgoing queue files
       reference articles by relative file path, offset, and size.

       It should be noted that dnewslink explicitly looks for the separator as
       a double-check against corruption.


DACTIVE.KP (/news/dactive.kp)
       The dactive.kp file is a key-token-pair database (see diablo-kp(5)).
       It is NOT compatible with the INN active file.  The dsyncgroups program
       with -G and -F flags may be used to create dactive.kp from an active
       file in "list active" format (see dsyncgroups(8)).  A dactive.kp file
       may also be created based on an active file on a remote host (also see
       dsyncgroups(8)).

       Diablo KP database files are human readable but should only be
       manipulated with the dkp program while Diablo is active.  If Diablo is
       inactive, KP files may be manipulated with dkp OR can be edited by
       hand.  Diablo's dactive.kp file serves the same purpose as INN's active
       file but, being a general token=value database, may contain additional
       information.  The intention is to use this database to track article
       number assigns and to store additional group description, moderator
       email, and PGP keys for the reader portion of Diablo.

       In Diablo, groups missing from dactive.kp do not normally effect the
       feeder side of the system.   However, Xref: headers are only generated
       for those newsgroups listed in dactive.kp.  This behavior can be
       changed through the 'activedrop' option in diablo.config

       The dactive.kp database is keyed off the group name and uses the
       following tokens, some of which are optional: NE (last stored article
       number, %010d format), NB (first stored article number %010d format), S
       (status), M (moderator email), and CPGP (pgp key, hierarchically
       recursive).  The status may contain multiple characters.  'n' indicates
       a disabled group, 'y' indicates an enabled group, 'm' indicates
       moderated.  itself is entirely optional (defaults to 'y').  The group
       description, if any, is stored with the GD key.

       When using the feeder to generate Xref: headers, the feeder creates a
       copy of the NE field called NX and uses that to track article number
       assignments.  This way both the reader and feeder can use the same
       physical active file when both the reader and feeder are running on the
       same box.  The dsyncgroups command has options to manipulate NE and NX.
       WARNING!  If NX is present, the reader will reset NE if NX < NE and an
       incoming article has been assigned a number less then NE.

       if a group is marked moderated, the moderator email is obtained via the
       M key.  If control messages related to the group (hierarchically
       speaking), the CPGP key contains the public key.  For example, there
       might be an entry for 'comp' with a CPGP key but since 'comp' is not a
       real newsgroup, the status would be blank.


SEE ALSO
       diablo(8), dsyncgroups(8), dicmd(8), didump(8), diload(8),
       dnewslink(8), doutq(8), dexpire(8), dexpireover(8), diconvhist(8),
       dilookup(8), dspoolout(8), dkp(8), dpath(8), diablo-files(5), diablo-
       kp(5)

                                                               DIABLO-FILES(5)