TOKE(1)                     General Commands Manual                    TOKE(1)

       toke - OpenBIOS tokenizer

       toke [options] forth-file

       toke is an open-source FCode Tokenizer from the OpenBIOS project.

       -h, -? Print a brief help message and then exit.

       -i, --ignore-errors
              Generate a Binary Output even if errors were reported.

       -l, --load-list
              Collect the names of floaded files into an FLoad-List file. The
              names collected are in the same form as they were presented in
              the fload statements.
              The name of the FLoad-List File is derived from the name of the
              binary output file, by replacing its extension with .fl , or, if
              the binary output file name had no extension, merely appending
              the extension .fl.
              The binary output file name used for this purpose is either the
              one specified on the command line, or the one created by

       -o, --output-name OutputFileName,
              Direct the binary output (FCode result of Tokenization) to the
              named file instead of to the default-named file. This option is
              not valid when multiple input files are named.

       -P, --dependencies
              Collect the fully-resolved pathnames of floaded and ENCODEd
              files into a dependency-list file.  The names collected are in
              the form that is presented to the host operating system: shell
              environment Variables and related expressions will be fully
              expanded, and the directory within the include-list in which the
              file was found will be attached.
              The name of the dependency-list file will be the same as that of
              the FLoad-list file, except that its extension will be .P
              instead of .fl.

       -v, --verbose
              Print additional messages (including advisories) during

       -I, --Include directory
              This tokenizer supports the notion of an include-list. The user
              creates the include-list by specifying a number of -I directory
              pairs on the command-line. All file-reads, whether for an fload
              command or an encode-file directive, will involve a search for
              the named file through the directories of the include-list, in
              the order they were supplied on the command-line.
              If no include-list is created, file-reads are relative to the
              current working Directory. If an include-list is created, file-
              reads are restricted to the directories within it.  For the
              current working directory to be included in the file-search, it
              must be specified explicitly.  -I.  will accomplish that quite

       -T, --Trace symbol
              This tokenizer supports the notion of a trace-list. The User
              creates the trace-list by specifying a number of -T symbol pairs
              on the command-line.
              When a name is defined, whether as an FCode, an alias, a Macro
              or anything else, either in normal tokenization mode or
              "Tokenizer Escape"‐mode, if it matches a symbol that has been
              added to the trace list, a trace note message will be issued
              indicating that a definition of that name has been created.
              Subsequent trace note Messages will be issued when the
              definition of that name is invoked.
              This trace-symbols feature can be helpful during maintenance of
              legacy code, for instance, when multiple symbols carry the same

       -d, --define Symbol[=Value]
              Define a command-line symbol and optionally, assign a value to
              it. If you wish the value to contain spaces or quotes, you can
              accomplish that using the shell escape conventions. This
              sequence may be repeated. Once a Symbol is defined on the
              command-line, it stays in effect for the duration of the entire
              batch of tokenizations (i.e., if there are multiple input files
              named on the command line). Command-line Symbols can be tested
              for purposes of conditional tokenization, or their assigned
              values can be evaluated.

       -f, --flag [no]<FlagName>
              The tokenizer recognizes a specific set of special-feature flag-
              names; each is associated with a specific non-standard variant
              behavior. Pass the flag-name as an argument to the -f switch to
              enable the behavior; to disable it, precede the flag-name with
              the optional string no.
              The settings of the special-feature flags can also be changed or
              displayed from within the source input file.
              The special-feature flags are all initially set to be enabled,
              except where noted.
              The flag-names and their associated special-features are as

                     Support IBM-style Local Values ("LV"s). Initially

                     Allow Semicolon for Local Values Separator ("Legacy").

                     Display a Warning Message when Semicolon is used as the
                     Local Values Separator.

                     Allow ABORT" macro.

                     ABORT" with implicit IF ... THEN

                     Use -2 THROW, rather than ABORT, in an Abort" phrase

                     Allow "\ (Quote-Backslash) to interrupt string parsing.

                     Allow \ (Backslash) to interrupt hex-sequence parsing
                     within a string.

                     Allow the C-style String-Escape pairs \n \t and \xx\ to
                     be treated as special characters in string parsing.

                     Override occurrences of the standard directive headerless
                     in the source with -- effectively -- headers to make all
                     definitions have a header. Occurrences of the directive
                     external will continue to behave in the standard manner.
                     Initially disabled.

                     All definitions will be made as though under the external
                     directive; occurrences of either Standard directive
                     headerless or headers in the source will be overridden.
                     This special-feature flag will also override the
                     Always-Headers special-feature flag in the event that
                     both have been specified. Initially disabled.

                     Display a WARNING message whenever a definition is made
                     whose name duplicates that of an existing definition.
                     Disabling this flag will suspend the duplicate-names test
                     globally, until it is re-enabled. A Directive is
                     supported that will suspend the test for the duration of
                     only a single definition, without affecting global

                     Display a WARNING message whenever an FCode function is
                     invoked that the Standard identifies as obsolete.

                     Issue advisory messages about the state of conditional
                     tokenization. (Remember that advisory messages are
                     displayed only if the verbose option -v is set.)
                     Initially disabled.

                     Lower-Case-Token-Names When outputting the names of
                     headered functions (token-names) to the binary output
                     file, override the character-case in which the names
                     appeared in the source, and convert them to Upper- or
                     Lower- ‐Case, respectively. (These flags do not affect
                     text string sequences, whose character-case is always
                     preserved.)  Initially disabled.

                     Save the Revision Level of the Vendor's ROM field of the
                     PCI Header in big-endian byte-order, rather than little-
                     endian as per the general PCI Standard convention. (This
                     flag does not affect any other field
                      of the PCI Header). Initially disabled.

                     Allow return-stack operations during interpretation.
                     While the standard specifies that usage of the operators
                     >r, r@, and r> while interpreting is allowed, actual
                     practice in the industry is inconsistent.  Developers who
                     wish to take a more cautious approach to this question
                     can disable this flag so that any attempt to use the
                     operators >r, r@, and r> in the interpreting state will
                     generate an ERROR Message.

       Also, the pseudo-flag-name help will cause a list of the flag-names and
       their associated special-features to be printed.


       Stefan Reinauer <>
       David L. Paktor <>

       This manual page was written by Aurelien Jarno <> for
       the Debian GNU/Linux system (but may be used by others).

                                 May 15, 2007                          TOKE(1)