vsl-query

VSL-QUERY(7)                                                      VSL-QUERY(7)



NAME
       vsl-query - Varnish VSL Query Expressions

OVERVIEW
       The Varnish VSL Query Expressions extracts transactions from the
       Varnish shared memory log, and perform queries on the transactions
       before reporting matches.

       A transaction is a set of log lines that belongs together, e.g. a
       client request or a backend request. The API monitors the log, and
       collects all log records that make up a transaction before reporting on
       that transaction. Transactions can also be grouped, meaning backend
       transactions are reported together with the client transaction that
       initiated it.

       A query is run on a group of transactions. A query expression is true
       if there is a log record within the group that satisfies the condition.
       It is false only if none of the log records satisfies the condition.
       Query expressions can be combined using boolean functions.  In addition
       to log records, it is possible to query transaction ids (vxid) in
       query.

GROUPING
       When grouping transactions, there is a hierarchy structure showing
       which transaction initiated what. The level increases by one on an
       'initiated by' relation, so for example a backend transaction will have
       one higher level than the client transaction that initiated it on a
       cache miss. Request restart transactions don't get their level
       increased to make it predictable.

       Levels start counting at 1, except when using raw where it will always
       be 0.

       The grouping modes are:

       · session

         All transactions initiated by a client connection are reported
         together. Client connections are open ended when using HTTP
         keep-alives, so it is undefined when the session will be reported. If
         the transaction timeout period is exceeded an incomplete session will
         be reported. Non-transactional data (vxid == 0) is not reported.

       · request

         Transactions are grouped by request, where the set will include the
         request itself as well as any backend requests or ESI-subrequests.
         Session data and non-transactional data (vxid == 0) is not reported.

       · vxid

         Transactions are not grouped, so each vxid is reported in its
         entirety. Sessions, requests, ESI-requests and backend requests are
         all reported individually. Non-transactional data is not reported
         (vxid == 0). This is the default.

       · raw

         Every log record will make up a transaction of its own. All data,
         including non-transactional data will be reported.

   Transaction Hierarchy
       Example transaction hierarchy using request grouping mode

          Lvl 1: Client request (cache miss)
            Lvl 2: Backend request
            Lvl 2: ESI subrequest (cache miss)
              Lvl 3: Backend request
              Lvl 3: Backend request (VCL restart)
              Lvl 3: ESI subrequest (cache miss)
                Lvl 4: Backend request
            Lvl 2: ESI subrequest (cache hit)

MEMORY USAGE
       The API will use pointers to shared memory log data as long as possible
       to keep memory usage at a minimum. But as the shared memory log is a
       ring buffer, data will get overwritten eventually, so the API creates
       local copies of referenced log data when varnishd comes close to
       overwriting still unreported content.

       This process avoids loss of log data in many scenarios, but it is not
       failsafe: Overruns where varnishd "overtakes" the log reader process in
       the ring buffer can still happen when API clients cannot keep up
       reading and/or copying, for instance due to output blocking.

       Though being unrelated to grouping in principle, copying of log data is
       particularly relevant for session grouping together with long lasting
       client connections - for this grouping, the logging API client process
       is likely to consume relevant amounts of memory. As the vxid grouping
       also logs (potentially long lasting) sessions, it is also likely to
       require memory for copies of log entries, but far less than session
       grouping.

QUERY LANGUAGE
       A query expression consists of record selection criteria, and
       optionally an operator and a value to match against the selected
       records.

          <record selection criteria> <operator> <operand>

       Additionally, a query expression can occur on the transaction itself
       rather than log records belonging to the transaction.

          vxid <numerical operator> <integer>

       A vxid query allows you to directly target a specific transacion, whose
       id can be obtained from an X-Varnish HTTP header, the default "guru
       meditation" error page, or Begin and Link log records.

       A query must fit on a single line, but it is possible to pass multiple
       queries at once, one query per line. Empty lines are ignored, and the
       list of queries is treated as if the 'or' operator was used to combine
       them.

       For example this list of queries:

          # catch varnish errors
          *Error

          # catch backend errors
          BerespStatus >= 500

       is identical to this query:

          (*Error) or (BerespStatus >= 500)

       Comments can be used and will be ignored, they start with the '#'
       character, which may be more useful when the query is read from a file.

       For very long queries that couldn't easily be split into multiple
       queries it is possible to break them into multiple lines with a
       backslash preceding an end of line.

       For example this query:

          BerespStatus >= 500

       is identical to this query:

          BerespStatus \
          >= \
          500

       A backslash-newline sequence doesn't continue a comment on the next
       line and isn't allowed in a quoted string.

   Record selection criteria
       The record selection criteria determines what kind records from the
       transaction group the expression applies to. Syntax:

          {level}taglist:record-prefix[field]

       Taglist is mandatory, the other components are optional.

       The level limits the expression to a transaction at that level. If left
       unspecified, the expression is applied to transactions at all levels.
       Level is a positive integer or zero. If level is followed by a '+'
       character, it expresses greater than or equal. If level is followed by
       a '-', it expresses less than or equal.

       The taglist is a comma-separated list of VSL record tags that this
       expression should be checked against. Each list element can be a tag
       name or a tag glob. Globs allow a '*' either in the beginning of the
       name or at the end, and will select all tags that match either the
       prefix or subscript. A single '*' will select all tags.

       The record prefix will further limit the matches to those records that
       has this prefix as their first part of the record content followed by a
       colon. The part of the log record matched against will then be limited
       to what follows the prefix and colon. This is useful when matching
       against specific HTTP headers. The record prefix matching is done case
       insensitive.

       The field will, if present, treat the log record as a white space
       separated list of fields, and only the nth part of the record will be
       matched against. Fields start counting at 1.

       An expression using only a record selection criteria will be true if
       there is any record in the transaction group that is selected by the
       criteria.

   Operators
       The following matching operators are available:

       · == != < <= > >=

         Numerical comparison. The record contents will be converted to either
         an integer or a float before comparison, depending on the type of the
         operand.

       · eq ne

         String comparison. 'eq' tests string equality, 'ne' tests for not
         equality.

       · ~ !~

         Regular expression matching. '~' is a positive match, '!~' is a
         non-match.

   Operand
       The operand is the value the selected records will be matched against.

       An operand can be quoted or unquoted. Quotes can be either single or
       double quotes, and for quoted operands a backslash can be used to
       escape the quotes.

       Unquoted operands can only consist of the following characters:

          a-z A-Z 0-9 + - _ . *

       The following types of operands are available:

       · Integer

         A number without any fractional part, valid for the numerical
         comparison operators. The integer type is used when the operand does
         not contain any period (.) characters.

       · Float

         A number with a fractional part, valid for the numerical comparison
         operators. The float type is used when the operand does contain a
         period (.) character.

       · String

         A sequence of characters, valid for the string equality operators.

       · Regular expression

         A PCRE regular expression. Valid for the regular expression
         operators.

   Boolean functions
       Query expressions can be linked together using boolean functions. The
       following are available, in decreasing precedence:

       · not <expr>

         Inverts the result of <expr>

       · <expr1> and <expr2>

         True only if both expr1 and expr2 are true

       · <expr1> or <expr2>

         True if either of expr1 or expr2 is true

       Expressions can be grouped using parenthesis.

QUERY EXPRESSION EXAMPLES
       · Transaction group contains a request URL that equals to "/foo"

            ReqURL eq "/foo"

       · Transaction group contains a request cookie header

            ReqHeader:cookie

       · Transaction group doesn't contain a request cookie header

            not ReqHeader:cookie

       · Client request where internal handling took more than 800ms.:

            Timestamp:Process[2] > 0.8

       · Transaction group contains a request user-agent header that contains
         "iPod" and the request delivery time exceeds 1 second

            ReqHeader:user-agent ~ "iPod" and Timestamp:Resp[2] > 1.

       · Transaction group contains a backend response status larger than or
         equal to 500

            BerespStatus >= 500

       · Transaction group contains a request response status of 304, but
         where the request did not contain an if-modified-since header

            RespStatus == 304 and not ReqHeader:if-modified-since

       · Transactions that have had backend failures or long delivery time on
         their ESI subrequests. (Assumes request grouping mode).

            BerespStatus >= 500 or {2+}Timestamp:Process[2] > 1.

       · Log non-transactional errors. (Assumes raw grouping mode).

            vxid == 0 and Error

HISTORY
       This document was written by Martin Blix Grydeland.

COPYRIGHT
       This document is licensed under the same licence as Varnish itself. See
       LICENCE for details.

       · Copyright (c) 2006 Verdens Gang AS

       · Copyright (c) 2006-2015 Varnish Software AS



                                                                  VSL-QUERY(7)