jigdo-file

JIGDO-FILE(1)                                                    JIGDO-FILE(1)



NAME
       jigdo-file - Prepare files for Jigsaw Download (distribution of huge
       files, e.g. CD images).

SYNOPSIS
       jigdo-file   COMMAND
        [ --image=cdrom.iso ] [ --jigdo=cdrom.jigdo ] [
       --template=cdrom.template ] [ --force ] [ MORE OPTIONS ] [ FILES ... |
       --files-from=f ]
        Common COMMANDs: make-template, make-image, verify


DESCRIPTION
       Jigsaw Download, or short jigdo, is a scheme developed primarily to
       make it easy to distribute huge filesystem images (e.g. CD (ISO9660) or
       DVD (UDF) images) over the internet, but it could also be used for
       other data which is awkward to handle due to its size, like audio/video
       files or large software packages.

       jigdo tries to ensure that the large file (always called image from now
       on) is downloaded in small parts which can be stored on different
       servers. People who want to download the image do so by telling the
       jigdo(1) (NOT IMPLEMENTED YET) download tool to process one `.jigdo'
       file; using it, jigdo downloads the parts and reassembles the image.
       jigdo-file is used to prepare the files for download.

       What makes jigdo special is that the parts that are used to reconstruct
       the image can have any size and content - they only need to be
       contained in a contiguous region anywhere in the image.

       For example, if you wish to distribute an ISO9660 image which contains
       a snapshot of an FTP server, you can instruct jigdo-file to prepare the
       download data in such a way that when people use jigdo to download the
       image, jigdo actually fetches the individual files from the FTP server
       and assembles them into an exact copy of your image - during the
       download! (If the image is not a filesystem dump, you can use split(1)
       to create the small parts that the image will be reassembled from.)

       You are completely free to choose where the individual parts of the
       image are stored: They may be in entirely different directories on
       different servers (e.g. because of storage/bandwidth constraints), but
       this is invisible to the people downloading your image. The information
       about available servers only needs to be added to the `.jigdo' file by
       you before distributing it.

       The `DETAILS' section below contains technical details on how jigdo
       works. The `EXAMPLES' section lists a number of common scenarios and
       may help you to get an idea of what jigdo is useful for.

OPTIONS
       Many options are specific to a particular COMMAND; the ones below are
       general or used by several commands. Further options are listed below
       with the individual commands. All options are silently ignored if they
       are not applicable to the current command. For any BYTES parameters to
       options, you can append one of the letters `k', `M' or `G' to the
       amount you specify, to indicate kilobytes, megabytes or gigabytes.

       -h --help
              Output short summary of commands and options.

       -H --help-all
              Output complete summary of commands and options.

       -v --version
              Output program version.

       -i --image=cdrom.iso
              Specify location of the file containing the image. The image is
              the large file that you want to distribute.

       -j --jigdo=cdrom.jigdo
              Specify location of the Jigsaw Download description file. The
              jigdo file is a human-readable file generated by jigdo-file, to
              which you add information about all the servers you are going to
              upload the files to.  jigdo will download this file as the first
              step of retrieving the image.

       -t --template=cdrom.template
              Specify location of the image `template' file. The template file
              is a binary file generated by jigdo-file, it contains
              information on how to reassemble the image and also (in
              compressed form) all the data from the image which was not found
              in any of the parts.

              Depending on the command, each of these three files is used
              sometimes for input, sometimes for output. If the file is to be
              used for output for a particular command and the output file
              already exists, jigdo-file exits with an error, unless --force
              is present.

              In most cases, you will only need to specify one out of -i -j
              -t, because any missing filenames will be deduced from the one
              you specify. This is done by first stripping any extension from
              the supplied name and then appending nothing (if deducing
              --image), `.jigdo' or `.template'.

       -r --report=default|noprogress|quiet|grep
              Control how verbose the program is, and what format the output
              has: noprogress is the same as default except that no `x% done'
              progress messages are printed.  quiet restricts the output to
              what is absolutely necessary, mostly error messages.  grep is
              only different from default for the make-template command: It
              enables output in a simple `<offset> <file>' format which is
              useful when searching for binary files in other binary files.

       -f --force
              Overwrite existent output files without complaining.

       --no-force
              This is the default. Refuse to overwrite existent output files.

       -c --cache=jigdo-cache.db
              jigdo-file usually needs to read the entire contents of all the
              FILES you specify. If you use it repeatedly (e.g. because you
              make a new CD image available daily), caching the file
              information will increase the program's speed significantly. The
              cache file is automatically created if it is not yet present.
              Data is usually both read from and written to it.

       --no-cache
              This is the default. Do not use a cache.

       --cache-expiry=SECONDS
              Set maximum age of cache entries. Any entries older than this
              will be removed from the cache. The default is 30 days. You can
              append one of the letters `h', `d', `w', `m', `y' to denote
              hours, days, weeks, months or years, respectively. A value of
              `0' or `off' disables expiry, so that all entries will stay in
              the cache forever. See the section `CACHE FILES' below for more
              information.

       --readbuffer=BYTES
              Set size of internal buffers. The default is 128k - if you have
              a fast disc, increasing this value may make jigdo-file faster,
              but in general, changing it is not necessary.

       --md5-block-size=BYTES
              Uninteresting internal parameter.  Set size of blocks into which
              files are subdivided. The default is 128k. If you change it, any
              cache file will have to be regenerated. Internally, jigdo-file
              may choose to use a slightly larger or smaller value.

       -T --files-from=file
              Read file and directory names from the specified file.  If file
              is `-', read names from standard input. Each line in the file is
              taken as a name, so the names may contain spaces, but not
              newline characters. An empty line causes jigdo-file to stop
              reading from the file.

              find(1) is a powerful tool for generating file lists, but make
              sure to use `find -type f' if possible - otherwise, if you
              instruct find to output both a filename and a symlink to that
              filename, jigdo-file will read the file contents twice.

       --hex  Output checksums in hexadecimal instead of Base64-like format.
              This should not be used with the make-template command, because
              the resulting `.jigdo' file violates the `.jigdo' file format.
              Its intended use is to make jigdo-file more interoperable with
              other Unix shell utilities like md5sum(1).

       --no-hex
              This is the default. Use jigdo's own Base64-like encoding of
              checksums.

       --debug[=help|=all|=UNIT,~UNIT... ]
              Switch on or off debugging output. Just `--debug' is equivalent
              to `--debug=all'. The argument is a comma-separated list of unit
              names for which debugging output is to be enabled, or disabled
              if the name is preceded by `~'. The special name `all' means all
              units. By default, debugging output is switched off except for
              the units `assert' and `general'. The exact list of available
              units for which debugging can be switched on depends on whether
              jigdo was compiled with debugging support - the list can be
              printed with `--debug=help'.

       FILES  Names of files or directories to use as input. These are the
              parts that are contained in the image. In case one of the names
              is a directory, the program recursively scans the directory and
              adds all files contained in it. While doing this, it follows
              symbolic links, but avoids symlink loops.

              If one of the filenames starts with the character `-', you must
              precede the list of files with `--'. A value of `-' has no
              special meaning in this list, it stands for a file whose name is
              a single hyphen.

COMMANDS
       The command name is the first non-option argument passed to jigdo-file.
       Most commands have short abbreviations as well as long names. The short
       command names should not be used in scripts - there may be incompatible
       changes to them in the future!

   MAKE-TEMPLATE, MT
       Reads image and FILES, creates `.jigdo' and `.template'. This is the
       main functionality of jigdo-file.

       It is possible to specify both --image=- and --files-from=-. In this
       case, first the list of files is read from standard input until an
       empty line is encountered. Everything following it is assumed to be the
       image data. This can be useful if you use mkisofs(1) or similar
       programs that can output the complete image on their standard output,
       because there is no need to store the image on disc temporarily.

       If a FILES argument contains the characters `//' (Unix) or `\.\'
       (Windows), this has special meaning. In the final jigdo file that users
       will download, each of the parts is referenced in the `[Parts]' section
       with a URI of the form `Label:some/filename'. (See `FORMAT OF .JIGDO
       FILES' below for a detailed description.) The `[Servers]' section gives
       a mapping of labels to servers on the internet, with lines like
       `Label=http://myserver.org/jigdofiles/'. Using this information, jigdo
       will create the final download URI for the part,
       `http://myserver.org/jigdofiles/some/filename'.  Specifying `//' (or
       `\.\') in a file or directory name serves to `cut off' the names at the
       right directory level. For example, if the Unix path of one of your
       FILES is `/path/some/filename', you can tell jigdo-file to cut off
       after the `/path' by passing it the argument `/path//some/filename', or
       `/path//' if you want the whole directory scanned. The path names need
       not be absolute; `somedirectory//' is also possible.

       --label Label=/path
              Specify a name to use as the label name for a path on disc.
              (Influences the output jigdo file.) If you used `//' in the
              FILES arguments as described above, jigdo-file will by default
              pick label names automatically (`A', `B' etc.). With this
              option, you can give labels more meaningful names. Note that the
              label name will only be used if one or more FILES begin with
              `/path//'.

              Try to use label names that start with uppercase characters, to
              disambiguate them clearly from protocol names like `http',
              `ftp'.

       --uri Label=http://some.server.org/
              By default, using --label as described above will cause lines of
              the form `Label=file:/path/' to be written to the `[Servers]'
              section of the output jigdo file. If you want to override the
              `file:' URI so that the line reads
              `Label=http://some.server.org/', you can do so by specifying
              --uri along with --label. Giving just --uri Label=... without
              the corresponding --label Label=... has no effect, and even if
              you specify both, an entry is only added to the `[Servers]'
              section if the label is referenced by at least one `[Parts]'
              entry.

              The supplied value is not quoted by the program; if it contains
              characters such as space or any of the characters #"'\ then you
              must quote it.  (Under Unix, you may need to quote the value
              twice to also protect it from the shell, e.g. \\\\ or '\\' to
              get a single backslash in the URI.)

              The mapping specified with an --uri option is ignored if it is
              already present in the output jigdo file.

              Users of the Windows version may notice that the `\' directory
              separators are converted into `/' in the `file:' URIs that are
              generated by default. This is done to increase cross-platform
              compatibility of `file:' - the print-missing command of the
              Windows version will automatically re-convert the characters
              when it prints the URIs. In case you supply your own `file:'
              URIs under Windows using --uri, you must also exchange `/' and
              `\'.

       -0 to -9
              Set amount of compression in the output template file, from -0
              (no compression) to -9 (maximum compression). The default is -9,
              which can make the template generation quite slow. By default,
              the compression algorithm used is the same as for gzip(1).

       --gzip and --bzip2
              Choose between the gzip and bzip2 compression algorithms. The
              default is gzip. Bzip2 usually gives a better compression ratio,
              but compression is significantly slower than with gzip.

       --min-length=BYTES
              Set minimum length of a part for jigdo-file to look for it in
              the image.  The default is 1k. Parts smaller than this will
              never be found in the image, so their data will be included in
              the template file. The search algorithm used requires such a
              minimum length, otherwise template generation could become
              extremely slow. If you know for sure that all your FILES are
              larger than a certain amount, you can increase jigdo-file's
              speed slightly by specifying the amount with this option.  There
              is a hard-wired absolute minimum of 256 bytes - anything lower
              will silently be set to 256.

       --merge=FILE
              Include the contents of FILE in the output `.jigdo' file. The
              file can contain data which you want added to the output (for
              example, a `[Servers]' section with a list of your servers as
              entries), or it can be the jigdo file output by an earlier run
              of jigdo-file.

              It is possible to specify the same file for input with --merge
              and for output with --jigdo. However, you will also need to use
              --force to make the program overwrite the old version of the
              jigdo file with the new one.  FILE can be `-' for standard
              input.

              When adding new information to the supplied file, jigdo-file
              will not insert new lines into the `[Parts]' section if an entry
              for the same MD5 checksum (but not necessarily with the same
              URI!) already exists, and it will not insert new lines into the
              `[Servers]' section if a completely identical entry already
              exists.

              When reading in the existing FILE, the behaviour is slightly
              different: The program preserves entries in the `[Parts]'
              section with identical checksum, but different URIs. For
              completely identical entries (same checksum and URI), only one
              entry is preserved and the duplicates are removed. The
              `[Servers]' section is left untouched.

       --image-section
              This is the default. Causes jigdo-file to add an `[Image]'
              section to the `.jigdo' file.

              As an exception, a new `[Image]' section is not added if you use
              --merge and the file to merge contains an `[Image]' section with
              a line which reads `Template-MD5Sum=' (end of line after the
              `='). In this case, the generated template data's MD5 checksum
              value is just added after the `=' of the first line of this form
              in the file - no whole new `[Image]' section is appended. This
              behaviour is useful because it allows you to pass via --merge an
              `[Image]' section with arbitrary content and then have the MD5
              checksum automatically added by jigdo-file. The section `FORMAT
              OF .JIGDO FILES' below explains the `[Image]' section contents
              in greater detail.

       --no-image-section
              Do not include an `[Image]' section in the `.jigdo' file. You
              need to add one yourself if you use this option. However, doing
              that is not easy (you also need to add a `Template-MD5Sum' line
              with the correct checksum, or jigdo will complain), so use of
              this option is discouraged.

       --servers-section
              This is the default. Causes jigdo-file to add a `[Servers]'
              section to the `.jigdo' file. This default section uses `file:'
              URIs, which allows for immediate reassembly of the image from
              the local filesystem, and is also useful if you want to edit the
              file manually and replace the `file:' URIs with other URIs.

       --no-servers-section
              Do not add a `[Servers]' section at the end of the `.jigdo'
              file. Useful e.g. if you are going to append the section with a
              script.

       --match-exec=SHELLCOMMAND
              Whenever a file is found in the image, execute the supplied
              command string by passing it to a shell.  jigdo-file sets up a
              number of environment variables with information about the file
              match. For example, if the file `/path//a/b/file' was found in
              the image and `Label:a/b/file' is going to be written to the
              `.jigdo' file:

              • LABEL="Label" - Name of the label for the file. The example
                assumes that `--label Label=/path' was specified by you.  In
                the absence of such an option, LABEL will be set but empty.

              • LABELPATH="/path/" - The path corresponding to the label, or
                in other words, the prefix of the matched file's path that
                will not appear in the output `.jigdo' file. Is set even
                without any `--label' option present.  Ends with a slash.

              • MATCHPATH="a/b/" - The rest of the path, without the leafname
                of the matched file. Is either empty or ends with a slash.

              • LEAF="file" - The leafname of the matched file.

              • MD5SUM="lNVdUSqbo2yqm33webrhnw" - The md5sum of the matched
                file, in Base64-like format.

              • FILE="/path//a/b/file" - For convenience, the complete path of
                the file. The variable is always set to
                $LABELPATH$MATCHPATH$LEAF.

       Please be careful to correctly quote the string passed to this option,
       otherwise your supplied command will not work with filenames that
       contain spaces. As an example, to create a backup of hard links to the
       matched files, use the following option: --match-exec='mkdir -p
       "${LABEL:-.}/$MATCHPATH" && ln -f "$FILE"
       "${LABEL:-.}/$MATCHPATH$LEAF"'

       By default, no command is executed. Use --match-exec="" to remove a
       command string which was set with an earlier use of this option.

       --greedy-matching
              This is the default. Imagine that your image contains a .tar
              file which in turn contains another file x, and that you provide
              both the .tar and the files inside it on the command line. When
              jigdo-file scans the image, it encounters the beginning of the
              .tar file, and then the file x.

              At this point, a decision must be made: Should the smaller file
              x be recorded as matched, or should it be ignored in favour of
              the larger (and thus better) match of the .tar file?
              Unfortunately, at this point it is not clear whether there will
              actually be a full match of the .tar, so by default, the program
              prefers the small match.

       --no-greedy-matching
              In the case where a large partial match is present and a shorter
              match has been confirmed, ignore the small match. (See the
              option above.)

   MAKE-IMAGE, MI
       Reads `.template' and FILES, creates image (or `imagename.tmp').
       Provides a rudimentary way of reassembling images - jigdo is usually
       better suited for this task. However, in contrast to jigdo, no `.jigdo'
       file is required.

       If the image is to be written to a file (and not to standard output),
       it is possible to create the image in several steps, with several
       invocations of `jigdo-file make-image', as follows: You first invoke
       jigdo-file, specifying as many files as are available at this time. The
       program scans the files, and those that are contained in the image are
       copied to a temporary file, whose name is formed by appending `.tmp' to
       the image filename.

       For all further files which could be parts of the image, you repeat
       this process. As soon as all parts are present, the temporary file will
       be truncated slightly (to delete some administrative data that jigdo-
       file appends at the end) and renamed to the final image name. The
       possibility of reassembling the image in several steps is especially
       useful for gathering files from removable media, e.g. several older
       CDs.

       Scripts using make-image can detect whether image creation is complete
       by checking the exit status: 0 signals successful creation, whereas 1
       means that more files need to be supplied. Other errors result in an
       exit status of 2 (`recoverable', e.g. file not found) or 3 (non-
       recoverable, e.g.  write error).

       --check-files
              This is the default. Whenever any part is copied to the image,
              re-check its checksum against the checksum stored in the
              template. It is recommended that you leave this switched on,
              even if it slows down image creation a bit.

       --no-check-files
              Do not check files' checksums when copying them to the image.
              This can be safely used when no cache file is used (which means
              that files will be written to the image immediately after being
              scanned) or the whole image is checked later with the verify
              command.

   PRINT-MISSING, PM
       Reads `.jigdo', `.template' and (if present) `imagename.tmp', outputs a
       list of URIs still needed to completely reassemble the image.

       Together with the make-image command, this provides most of the
       functionality of jigdo on the command line.

       For each part that is not yet present in the temporary image file, the
       file checksum is looked up in the `[Parts]' section of the jigdo file.
       Any label in the corresponding entry is then expanded according to the
       label definitions in the `[Servers]' section and printed on standard
       output. jigdo allows you to specify several alternative locations for
       each label in this section, but print-missing will only output the
       first one for each missing part.

       If the checksum cannot be found in the `[Parts]' section (this Should
       Not Happen unless you deleted that section), a lookup is instead made
       for `MD5Sum:<checksum>', just like with jigdo. (Thus, if you want to
       get rid of the `[Parts]' section, you can do so if you rename each part
       to its own checksum.)

       --uri Label=http://some.server.org/
              Override the entries in the `.jigdo' file for any label with a
              URI of your choice. With the example above, a `[Parts]' entry of
              `Label:some/filename' will cause the line
              `http://some.server.org/some/filename' to be printed.

              The supplied value is not quoted by the program; if it contains
              characters such as space or any of the characters #"'\ then you
              must quote it.  (Under Unix, you may need to quote the value
              twice to also protect it from the shell, e.g. \\\\ or '\\' to
              get a single backslash in the URI.)

   PRINT-MISSING-ALL, PMA
       Just like print-missing, this command outputs a list of URIs still
       needed to completely reassemble the image. However, all alternative
       download locations are printed instead of just one. In the output, the
       URIs for a file are separated from other files' URIs with blank lines.
       The --uri option has the same effect as for print-missing.

   VERIFY, VER
       Reads image (presumably generated with make-image) and `.template',
       checks for correct checksum of image.

       The template data does not only contain checksums of the individual
       parts, but also of the image as a whole.  make-image already performs a
       number of internal checks, but if you like, you can additionally check
       the image with this command.

   SCAN, SC
       Reads all the FILES and enters them into the cache, unless they are
       already cached. The --cache option must be present for this command.

       --no-scan-whole-file
              This is the default. This only causes the first --md5-block-size
              bytes of each file to be read. If the cache is used later by
              jigdo-file make-image, the rest of the file will be read once
              these first bytes are recognized in the input image.

       --scan-whole-file
              Immediately read the entire file contents and store them in the
              cache.

   MD5SUM, MD5
       Reads all the FILES and prints out MD5 checksums of their contents.
       This command is quite similar to md5sum(1), except that the checksum is
       output in the Base64-like encoding which is also used elsewhere by
       jigdo-file.

       The FILES arguments are processed in the same way as with the other
       commands, which means that recursion automatically takes place for any
       arguments that are directories, and that symbolic links are not listed
       except when the file(s) they point to are not reachable directly.

       In the checksum list printed on standard output, only the part of the
       filename following any `//' (or `\.\' on Windows) is printed. Any
       --cache will be used for querying files' MD5 checksums and/or writing
       the checksums of scanned files.

   LIST-TEMPLATE, LS
       Reads a `.template' file and outputs low-level information about the
       image and all parts contained in it, including offset, length and
       checksum.

       You can also use this command with temporary image files (by specifying
       something like --template=imagename.tmp) - in that case, the output
       also distinguishes between parts that have been written to the image
       and parts that haven't.

       The exact output format may change incompatibly between different jigdo
       releases. The following different types of lines can be output. `have-
       file' only occurs for `.tmp' files, indicating a file that has already
       been successfully written to the temporary file:

       in-template offset-in-image length
       need-file offset-in-image length file-md5sum filestart-rsyncsum
       have-file offset-in-image length file-md5sum filestart-rsyncsum
       image-info image-length image-md5sum rsyncsum-size

DETAILS
       Jigsaw Download was created with the format of ISO9660 CD images in
       mind - however, the following also applies to many other filesystem
       formats, as well as to `tar' archives and uncompressed `zip' archives.
       A CD image contains both information for organizing the filesystem
       (header with disc name etc., ISO9660 directory data, data of extensions
       such as Joliet or RockRidge, zero padding) and the files contained on
       the CD. An important property that jigdo relies on is that each file is
       stored in one contiguous section of the image; it is not split into two
       or more parts.

       When jigdo-file is given a number of files that might be contained in
       an image, it detects whether any of the files are present using a
       `rolling checksum' inspired by the one used by rsync(1). The resulting
       data is written to the `.template' file: If a section of the image
       could not be matched (e.g. it was directory information), the data is
       compressed and written directly to the template. However, if a matching
       file was found, its data is omitted from the template. Instead, only a
       reference (an MD5 checksum of the file) is inserted in the template.

       Note that the template data only contains binary data, it does not
       contain any filenames or URIs, since it cannot be easily edited in case
       any of these values need to be changed. All that information is stored
       in the `.jigdo' file, a text file to which you can add URLs for your
       server(s). The jigdo file provides a mapping for each MD5 checksum to
       one or more alternative download locations for the corresponding part.

       Apart from the mapping of MD5 sums to URIs, the jigdo file also
       contains an URI pointing to a download location for the template file.
       This way, the jigdo download tool only needs to be given one URI (that
       of the `.jigdo' file) to be able to download and reassemble the
       complete image.

FORMAT OF .JIGDO FILES
       The overall format of `.jigdo' files follows that of `.ini' files, as
       also used by the Gnome and KDE projects for some data. The file is
       organized into sections, each of which is preceded by a line reading
       `[Sectionname]'. Within each section, lines have the form
       `Label=Value'. Such lines are also called `entries' below. All `.jigdo'
       files use UTF-8 as their character encoding.

       Comments are introduced with the `#' character and extend to the end of
       the line. Whitespace is ignored at line start and end as well as to the
       left and right of section names and the `=' in entries. Furthermore,
       the jigdo utilities split up the text of the entry value (i.e. the part
       after the `=') into whitespace-separated words, much like the Unix
       shell. Single '' and double "" quotes can be used to prevent that e.g.
       URIs containing whitespace are split apart. Similarly, characters with
       special meaning (the characters '"#\ and space/tab) must be quoted with
       \ to appear in the value. As with the shell, there is a difference
       between ' ' and " ": Within ' ', the characters "#\ and whitespace lose
       their special meaning and become ordinary characters, whereas within
       " ", only the characters '# and whitespace lose their special meaning -
       in other words, backslash escapes still work inside " ", but not ' '.

       `.jigdo' files can optionally be compressed with gzip(1). jigdo-file
       always outputs uncompressed files, which you can compress yourself.
       jigdo-lite supports single uncompressed and compressed files.

       (Behaviour which may change in the future and which should not be
       relied upon: jigdo additionally supports any number of concatenated
       plaintext and gzipped parts in the files - for example, you can
       compress a `.jigdo' file and then add a couple of lines of uncompressed
       data to the end.)

       In all cases, the `.gz' extension should be removed from the filename -
       the tools will determine automatically from the file contents whether a
       file is compressed or not.

       Below is a description of the individual section names used by jigdo.

   JIGDO SECTION
       [Jigdo]
       Version=1.1
       Generator=jigdo-file/1.0.0

       Information about the version of the jigdo file format used, and the
       program that generated it. There should be one such section per
       `.jigdo' file.

   IMAGE SECTION
       [Image]
       Filename="filename for saving on user's disc"
       Template="URI where to fetch template file"
       Template-MD5Sum=OQ8riqT1BuyzsrT9964A7g
       ShortInfo=single-line description of the image (200 characters max.)
       Info=long description (5000 characters max.)

       The value for the `Template' entry can be either an URL (absolute or
       relative to the URL of the jigdo file) or a string of the form
       `Label:pathname' (UNIMPLEMENTED), as described below.

       The `Template-MD5Sum' entry is added by jigdo-file and specifies the
       MD5 checksum of the generated `.template' file. It is used by jigdo to
       detect cases where the downloaded template data is corrupted or belongs
       to a different image.

       Unlike other entry values, the values of the `ShortInfo' and `Info'
       entries are not split up into words, instead all quoting is preserved.

       The value of the `Info' entry is special in that jigdo(1) can
       optionally parse XML markup it contains. If the markup has errors such
       as unbalanced/unsupported tags, the string is displayed literally,
       without XML parsing. Supported tags are <b></b> (bold), <i></i>
       (italic), <tt></tt> (typewriter font), <u></u> (underline), <big></big>
       (larger font), <small></small> (smaller font) and <br/> (linebreak).
       Supported entities include &lt; (`<'), &gt; (`>') and &amp; (`&'). Note
       that the whole `Info' entry must be on one line in the jigdo file.

       This section may occur multiple times, but all except the first one
       will be ignored. This is useful e.g. when creating a `.jigdo' file for
       a DVD image when you already have `.jigdo' files for CDs with the same
       content: You can simply `[Include]' (see below) the CDs' jigdo files at
       the end of the DVD jigdo file, after its `[Image]' section.

   PARTS SECTION
       [Parts]
       xJNkjrq8NYMraeGavUpllw=LabelA:part0
       GoTResP2EC6Lb_2wTsqOoQ=LabelA:part1
       kyfebwu6clbYqqWUdFIyaw=LabelB:some/path/part2
       -J9UAimo0Bqg9c0oOXI1mQ=http://some.where.com/part3

       All lines in the section, which provides the mapping from MD5 checksums
       to URIs, have the same format: On the left side of the `=' the checksum
       (encoded with a Base64-like encoding) is given, and on the right a
       string corresponding to the part with this checksum; either a complete
       URI or a string of the form `Label:pathname', which is expanded into
       one or more URIs by looking up the definition(s) for the Label in the
       `[Servers]' section.

       In case a particular MD5 checksum cannot be found in any `[Parts]'
       section by jigdo, the program will perform a lookup for
       `MD5Sum:<checksum>', e.g. for `MD5Sum:xJNkjrq8NYMraeGavUpllw' if you
       deleted the line for `part0' above.

       A checksum appearing multiple times in this section indicates
       alternative download locations for the part.

       There may be any number of `[Parts]' sections in the file; they are all
       considered when looking up MD5 checksums.

       jigdo-file always puts the `[Parts]' section at the end of the file,
       and it even rearranges any file specified with --merge to have only one
       such section at the end. This is done to allow jigdo to display the
       information from the `[Image]' section while the rest of that file is
       still being downloaded.

   SERVERS SECTION
       [Servers]
       LabelA=http://myserver.org/
       LabelA=ftp://mirror.myserver.org/
       LabelB=LabelC:subdirectory/
       LabelC=http://some.where.com/jigdo/

       All lines in the section, which provides the mapping from server labels
       to server locations, have the same format: On the left side of the `='
       the label name is given, and on the right the value to expand the label
       name to.

       A label name appearing multiple times in this section indicates
       alternative download locations for the parts that use the label in the
       `[Parts]' section. This notation makes it very easy to add mirrors to
       the jigdo file.

       As shown by the example above, the label values may themselves
       reference other labels. In this case, the entry
       `LabelB:some/path/part2' in the `[Parts]' section will expand to
       `http://some.where.com/jigdo/subdirectory/some/path/part2'.  Loops in
       the label definitions result in undefined behaviour and must be
       avoided.

       There may be any number of `[Servers]' sections in the file; they are
       all considered when looking up labels. Either of `[Parts]' or
       `[Servers]', but not both, can be omitted from the jigdo file.

   COMMENT SECTION
       [Comment]
       Any text, except that lines must not begin with `['.

       All text following a `[Comment]' or `[comment]' line is ignored, up to
       the next line with a section label.

   INCLUDE DIRECTIVE
       [Include http://some.url/file.jigdo]

       Lines of this form cause the content of the specified jigdo file to be
       downloaded and parsed just like the main jigdo file. The effect will be
       the same as copying the included file's contents into the file which
       contains the include directive. (Exception: Any relative URLs are
       always resolved using the URL of the `.jigdo' file that contains that
       relative URL.)

       The URL argument can be an absolute or relative URL.  Relative URLs are
       assumed to be relative to the URL of the jigdo file which contains the
       include directive. Includes can be nested, but it is an error to create
       a loop of include directives. It is not possible to use URLs of the
       form `Label:pathname'.

       The URL cannot be quoted with "". Any `]' characters in the argument
       must be escaped as `%5D', and any spaces as `%20'.

       Include directives are only supported by jigdo, they are ignored by
       jigdo-lite.

       An include directive terminates any previous section, but it does not
       start a new one. In other words, a new section must always be started
       after the include line, jigdo does not allow normal entries to appear
       below the `[Include]'.

CACHE FILES
       Any file specified with the --cache option is used to store information
       about the FILES presented to jigdo-file. When querying the cache, a
       file is considered unchanged (and the cached data is used) only if
       filename, file size and last modification time (mtime) match exactly.
       For the filename match, not the entire file name is used, but only the
       part following any `//', so that any changes to the part before the
       `//' will not invalidate the cache.

       Old cache entries are removed from the cache if they have not been read
       from or written to for the amount of time specified with --cache-
       expiry. Entries are not immediately removed from the cache if the file
       they refer to no longer exists - this makes it possible to cache
       information about files on removable media.

       Cache expiry only takes place after jigdo-file has done its main work -
       if any old entries are accessed before expiry takes place, they will be
       kept.  For example, if the program is run using the default expiry time
       of 30 days, but accesses a cache file with entries generated 2 months
       ago, then entries in that cache will be considered, and only those
       cache entries that were not needed during the program run will be
       expired.

       Due to a peculiarity of the underlying database library (libdb3), cache
       files never shrink, they only grow. If a large number of entries was
       expired from your cache file and you want it to shrink, you can either
       just delete it (of course then everything will have to be regenerated)
       or use the utilities accompanying libdb3 to dump and restore the
       database, with a command like `db3_dump old-cache.db | db3_load new-
       cache.db'. For Debian, these programs are supplied in the package
       `libdb3-util'.

       If a different --md5-block-size is specified, the entire file needs to
       be re-read to update its cache entry. If a different --min-length is
       specified, only the first `md5-block-size' bytes of the file need to be
       re-read.

EXAMPLES
   PREPARING YOUR CD IMAGE FOR DISTRIBUTION
       You have created a CD image `image.iso' from some of the files stored
       in the directory `/home/ftp' on your harddisc, which is also available
       online as `ftp://mysite.org'.  As you don't want to waste space by
       effectively hosting the same data twice (once as files on the FTP
       server, once inside the image), and you are fed up with users'
       downloads aborting after 200MB and their restarting the download dozens
       of times, you decide to use jigdo. How do you prepare the image for
       download?

       In fact, only one command is necessary:


              jigdo-file make-template --image=image.iso
              --jigdo=/home/ftp/image.jigdo
              --template=/home/ftp/image.template /home/ftp// --label
              Mysite=/home/ftp --uri Mysite=ftp://mysite.org/

       People can now point jigdo at `ftp://mysite.org/image.jigdo' to
       download your image. The template file needs to be accessible as
       `ftp://mysite.org/image.template'.

       Note that nothing prevents you from doing the same for an FTP server
       that isn't administrated by you - in that case, you only need to host
       the `.jigdo' and `.template' files on your own server/homepage.

   PREPARING AN ARBITRARY LARGE FILE FOR DISTRIBUTION
       We assume that you have a large file that is not a filesystem, e.g.
       `movie.mpeg'. Because of space problems, you want to distribute the
       data on two servers.

       In this case, the parts of the image need to be generated artificially
       with the split command. For example, to create chunks of 4MB each, use
       `split -b 4m movie.mpeg part'. Copy the resulting files `partXX' into
       two directories `1' and `2' that you create, according to how you want
       the files distributed between the servers. Next, create the jigdo and
       template files with `jigdo-file make-template --image=movie.mpeg 1//
       2//'. You will need to edit the `.jigdo' file and provide the right
       URIs for the two servers that you are going to upload the `partXX'
       files to.

   CUSTOMIZED VERSIONS OF IMAGES
       Because it is possible to assign a different URI for each part of an
       image if necessary, jigdo is very flexible. Only one example is the
       possibility of customized versions of images: Suppose that someone is
       distributing a CD image, and that you want to make a few small changes
       to it and redistribute your own version. You download the
       `official.iso' CD image with jigdo (passing it the URL of
       `official.jigdo'), write it to CD-R, make your changes (say, adding
       files from the `myfiles' directory on your harddisc) and produce your
       own version, `myversion.iso'.  Next, you instruct jigdo-file to create
       the jigdo and template files for your modified image, using the command


              jigdo-file make-template --image=myversion.iso /mnt/cdrom/
              myfiles// --label My=myfiles/ --uri My=http://my.homepage.net/
              --merge=official.jigdo
       while `official.iso' is mounted under `/mnt/cdrom'. By using --merge,
       you have told jigdo-file to take the contents of `official.jigdo', add
       to it a new `[Image]' section for `myversion.iso' and write the
       resulting jigdo file to `myversion.jigdo' - so now `myversion.jigdo'
       offers two images for download, the original version and your modified
       version. (If you do not want it to offer the official version, edit it
       and remove the `[Image]' section that lists `official.iso'.)

       Now you can upload the `.jigdo' file, the `.template' file and also the
       files in `myfiles' to `http://my.homepage.net/'.  Thus, for people to
       download your modified image, you do not need to upload the complete
       image contents to your web space, but only the changes you made!

       (In case you only made very few changes, you could also omit the
       `myfiles' parameter in the command above, then all your changes end up
       in the new template file.)

   COMBINING MANY JIGDO-MANAGED IMAGES INTO ONE
       It is also no problem to combine data from several sources that use
       jigdo. For example, if of five different and unrelated servers each one
       distributes a different CD image via jigdo, you can create a customized
       DVD image that contains the data from all these CDs. When people use
       jigdo to download your image, the individual files on the DVD are
       fetched from the same sources as the original CDs.

       Consequently, even though you will be distributing a 3.2GB file via
       your web space, the actual amount of data that is stored on your server
       will only be in the order of several MBs.

BUGS
       For certain contents of one of the input files, most notably a sequence
       of zero bytes longer than --min-length at the start of the file and an
       area of zeros preceding the file data in the image, jigdo-file make-
       template may fail to find the file in the image. Unfortunately, this
       restriction cannot be avoided because the program could become very
       slow otherwise. If you use the --debug option, all instances of jigdo-
       file discarding possible matches are indicated by lines containing the
       word `DROPPED'.

       In fact, not only all-zeroes files trigger this behaviour, but also
       files which contain at their start a long sequence of short identical
       strings. For example, both a file containing only `a' characters and
       one containing `abcabcabcabc...' are problematic.

SEE ALSO
       jigdo(1) (NOT YET IMPLEMENTED), jigdo-lite(1), jigdo-mirror(1),
       split(1) (or `info split'), find(1) (or `info find'), mkisofs(1),
       md5sum(1)

AUTHOR
       Jigsaw Download <URL:http://atterer.org/jigdo/> was written by Richard
       Atterer <jigdo atterer.org>, to make downloading of CD ROM images for
       the Debian Linux distribution more convenient.



                                  19 May 2006                    JIGDO-FILE(1)