Add docs and examples for req-disposition.

Closes #24

docs/ref-match-flags.md

+# ``match-flags`` -- configuring match operations
+
+This is the authoritative reference for the ``match-flags`` field,
+which is used in a number of places in the custom resource
+configurations where string comparisons are specified. For example in
+the [``rewrites`` configuration](/docs/ref-varnish-cfg.md) for header
+and URL rewriting, and in the [``req-disposition``
+configuration](/docs/ref-req-disposition.md) to specify the
+disposition of client requests.
+
+Three types of string comparison are used by this Ingress
+implementation:
+
+* regular expression matching: strings are matched against patterns
+  with the syntax and semantics of [RE2 regular
+  expressions](https://github.com/google/re2/wiki/Syntax)
+
+* fixed string matches: strings are tested for equality with literal
+  string values. Characters such as wildcards or regular expression
+  metacharacters have no special meaning.
+
+* prefix matches: strings are tested as to whether they begin with any
+  one of a set of strings. These are also literal matches against the
+  prefixes, in the sense that no character in the prefix has a special
+  meaning.
+
+The string comparisons are typically configured for an array of
+possible string values. The comparisons are evaluated as true if they
+are true for any one of the strings in the array. In other words, the
+string array represents a boolean OR of comparisons.
+
+The ``match-flags`` object is used to configure and control the
+comparison operations. ``match-flags`` is optional in every context in
+which it may be specified -- if it is left out, then comparisons are
+executed with default options.
+
+Only the ``case-sensitive`` field may be set for fixed string and
+prefix matches, typically configured with the string enum values
+``equal`` and ``prefix``, respectively.  All of the other fields are
+permitted only for regular expression matches, typically configured as
+``match``. In other words, case insensitivity can be specified for all
+comparison operations, but the other fields apply only to regex
+matching.
+
+The ``match-flag`` fields are adapted from the
+[RE2](https://github.com/google/re2/) library (via [VMOD
+re2](https://code.uplex.de/uplex-varnish/libvmod-re2)).  The fields
+are:
+
+* ``case-sensitive`` (default ``true``): if ``false``, then
+  comparisons (regex, fixed string or prefix) are case insensitive.
+
+* ``anchor`` (default ``none``): sets anchoring at start-of-string or
+  end-of-string for every pattern to be matched; equivalent to using
+  the ``^`` and ``$`` for start- and end-of-string in the notation for
+  each pattern. Possible values are:
+
+    * ``start``: each pattern is anchored at the start
+
+    * ``both``: each pattern is anchored at both start and end.
+
+    * ``none`` (default): no implicit anchoring (but ``^`` and/or
+      ``$`` may be used in individual patterns)
+
+* ``literal`` (default ``false``): if ``true``, then the strings to to
+  be matched are matched literally, with no special meaning for regex
+  metacharacters (despite the use of regex matching).
+
+* ``never-capture`` (default ``false``): if ``true``, then substring
+  capturing is not executed for regex matches. Capturing
+  backreferences is necessary for some applications, such as header
+  and URL rewrites.  But consider setting ``never-capture`` to
+  ``true`` if your patterns have round parentheses ``()`` for grouping
+  only, and backreferences are not needed, since regex matches are
+  faster without the captures.
+
+* ``utf8`` (default ``false``): if ``true``, then characters in each
+  pattern match UTF8 code points; otherwise, the patterns and the
+  strings to be matched are interpreted as Latin-1 (ISO-8859-1). Note
+  that characters in header values and URL paths almost always fall in
+  the ASCII range, so the default is usually sufficient. Note also that
+  this differs from the default in the RE2 library.
+
+* ``longest-match`` (default ``false``): if ``true``, then the matcher
+  searches for the longest possible match where alternatives are
+  possible. For example with the pattern ``a(b|bb)`` and the string
+  ``abb``, ``abb`` matches when ``longest-match`` is ``true``, and
+  backref 1 is ``bb``. Otherwise, ``ab`` matches, and backref 1 is
+  ``b``.
+
+* ``posix-syntax`` (default ``false``): if ``true``, then patterns are
+  restricted to POSIX (egrep) syntax. Otherwise, the full range of
+  [RE2](https://github.com/google/re2/wiki/Syntax) is available.
+
+    The next two flags (``perl-classes`` and ``word-boundary``) are
+    only consulted when ``posix-syntax`` is ``true``.
+
+* ``perl-classes`` (default ``false``): if ``true`` and
+  ``posix-syntax`` is also ``true``, then the perl character classes
+  ``\d``, ``\s``, ``\w``, ``\D``, ``\S`` and ``\W`` are permitted in a
+  pattern. When ``posix-syntax`` is ``false``, the perl classes are
+  always permitted.
+
+* ``word-boundary`` (default ``false``): if ``true`` and
+  ``posix-syntax`` is also ``true``, then the perl assertions ``\b``
+  and ``\B`` (word boundary and not a word boundary) are permitted in
+  a pattern. When ``posix-syntax`` is ``false``, the word boundary
+  assertions are always permitted.
+
+* ``max-mem`` (integer, default 8MB): an upper bound (in bytes) for
+  the size of the compiled pattern. If ``max-mem`` is too small, the
+  matcher may fall back to less efficient algorithms, or the pattern
+  may fail to compile.
+
+    This field very rarely needs to be set; the default is the RE2
+    default, and is sufficient for typical patterns. Increasing
+    ``max-mem`` is usually only necessary if VCL loads fail due to
+    failed regex compiles, and the error message (shown in Event
+    notifications and the controller log) indicates that the pattern
+    is too large.
+
+See the [``examples/``](/examples/) folder, particular the examples
+for [rewriting](/examples/rewrite) and [client request
+disposition](/examples/req-disposition/), for working examples in
+which the ``match-flags`` object is used.

docs/ref-varnish-cfg.md

@@ -779,87 +779,10 @@ Elements of the ``rewrites`` array may have these fields, of which
     that a synthetic response with status 503 and the reason "VCL
     failed" is returned for the request.
 
-* ``match-flags`` is an object with configuration to control comparison
-  operations. If ``match-flags`` is absent, then comparisons are executed
-  with default options.
-
-    Only the ``case-sensitive`` field may be set if ``compare`` is
-    ``equal`` or ``prefix``; all of the other fields are permitted
-    only if ``compare`` is ``match``. In other words, case
-    insensitivity can be specified for all comparison operations, but
-    the other fields apply only to regex matching. The fields are:
-
-    * ``case-sensitive`` (default ``true``): if ``false``, then regex
-      and fixed-string comparisons are case insensitive.
-
-    * ``anchor`` (default ``none``): sets anchoring at start-of-string
-      or end-of-string for every pattern in the ``rules`` array;
-      equivalent to using the ``^`` and ``$`` for start- and
-      end-of-string in the notation for each pattern. Possible values
-      are:
-
-        * ``start``: each pattern is anchored at the start
-
-        * ``both``: each pattern is anchored at both start and end.
-
-        * ``none`` (default): no implicit anchoring (but ``^`` and/or
-          ``$`` may be used in individual patterns)
-
-    * ``literal`` (default ``false``): if ``true``, then the strings
-      in the ``value`` fields of the ``rules`` are matched literally,
-      with no special meaning for regex metacharacters.
-
-    * ``never-capture`` (default ``false``): if ``true``, then
-      substring capturing is not executed for regex matches. Consider
-      setting ``never-capture`` to ``true`` if your patterns have
-      round parentheses ``()`` for grouping only, and backreferences
-      are not used in rewrite strings, since regex matches are faster
-      without the captures.
-
-    * ``utf8`` (default ``false``): if ``true``, then characters in
-      each pattern match UTF8 code points; otherwise, the patterns and
-      the strings to be matched are interpreted as Latin-1
-      (ISO-8859-1). Note that characters in header values and URL
-      paths almost always fall in the ASCII range, so the default is
-      usually sufficient.
-
-    * ``longest-match`` (default ``false``): if ``true``, then the
-      matcher searches for the longest possible match where
-      alternatives are possible. For example with the pattern
-      ``a(b|bb)`` and the string ``abb``, ``abb`` matches when
-      ``longest-match`` is ``true``, and backref 1 is
-      ``bb``. Otherwise, ``ab`` matches, and backref 1 is ``b``.
-
-    * ``posix-syntax`` (default ``false``): if ``true``, then patterns
-      are restricted to POSIX (egrep) syntax. Otherwise, the full
-      range of [RE2](https://github.com/google/re2/wiki/Syntax) is
-      available. The next two flags (``perl-classes`` and
-      ``word-boundary``) are only consulted when ``posix-syntax`` is
-      ``true``.
-
-    * ``perl-classes`` (default ``false``): if ``true`` and
-      ``posix-syntax`` is also ``true``, then the perl character
-      classes ``\d``, ``\s``, ``\w``, ``\D``, ``\S`` and ``\W`` are
-      permitted in a pattern. When ``posix-syntax`` is ``false``, the
-      perl classes are always permitted.
-
-    * ``word-boundary`` (default ``false``): if ``true`` and
-      ``posix-syntax`` is also ``true``, then the perl assertions
-      ``\b`` and ``\B`` (word boundary and not a word boundary) are
-      permitted in a pattern. When ``posix-syntax`` is ``false``, the
-      word boundary assertions are always permitted.
-
-    * ``max-mem`` (integer, default 8MB): an upper bound (in bytes)
-      for the size of the compiled pattern. If ``max-mem`` is too
-      small, the matcher may fall back to less efficient algorithms,
-      or the pattern may fail to compile.
-
-        This field very rarely needs to be set; the default is the RE2
-        default, and is sufficient for typical patterns. Increasing
-        ``max-mem`` is usually only necessary if VCL loads fail due to
-        failed regex compiles, and the error message (shown in Event
-        notifications and the controller log) indicates that the
-        pattern is too large.
+* ``match-flags`` is an object with configuration to control
+  comparison operations. If ``match-flags`` is absent, then
+  comparisons are executed with default options. See the
+  [``match-flags`` reference](/docs/ref-match-flags.md) for details.
 
 * ``vcl-sub`` is an enum indicating the
   [VCL subroutine](https://varnish-cache.org/docs/6.1/reference/states.html)
@@ -914,3 +837,11 @@ Elements of the ``rewrites`` array may have these fields, of which
     If more than one rewrite in the ``rewrites`` array specifies the
     same VCL subroutine, then they are executed in that subroutine in
     the order in which they appear in the array.
+
+## ``spec.req-disposition``
+
+The ``req-disposition`` is optional, and if present contains a
+configuration of the disposition of client requests -- how client
+requests are processed further after request headers have been
+received. See the [``req-disposition``
+reference](/docs/ref-req-disposition.md) for details.

examples/README.md

@@ -29,6 +29,19 @@ requirements.
 * Specifying [rewrite rules](/examples/rewrite) for request headers,
   response headers, and URL paths.
 
+* Specifying the [disposition of client requests](/example/req-disposition).
+  This can enable a number of features, such as:
+
+    * allowing caching for requests that use cookies or basic
+      authentication.
+
+    * defining cacheability based on properties of the client request,
+      such as URL path patterns.
+
+    * defining white- and blacklists for requests.
+
+    * defining the means to purge cache entries via a request.
+
 * The [BackendConfig](/examples/backend-config) Custom Resource, to
   configure properties such as timeouts, health probes and
   load-balancing for Services to which requests are routed according

examples/req-disposition/alt-builtin.yaml

@@ -13,8 +13,11 @@ spec:
   services:
     - varnish-ingress
 
+  # Configure the disposition of client requests.
   req-disposition:
 
+    # Requests for HTTP/1.1. without a Host header are rejected with
+    # 400 Bad Request, as in builtin.yaml and built-in vcl_recv.
     - conditions:
       - comparand: req.http.Host
         compare: not-exists
@@ -30,6 +33,10 @@ spec:
         action: synth
         status: 400
 
+    # Requests with the CONNECT method invoke pipe mode. This may be
+    # appropriate for a WebSockets application. If you don't need pipe
+    # mode for any purpose, just include CONNECT in the array of
+    # request method names in the next stanza.
     - conditions:
       - comparand: req.method
         compare: equal
@@ -38,6 +45,8 @@ spec:
       disposition:
         action: pipe
 
+    # Requests with any non-standard method get a synthetic 405 Method
+    # Not Allowed response.
     - conditions:
       - comparand: req.method
         compare: not-equal
@@ -54,6 +63,8 @@ spec:
         action: synth
         status: 405
 
+    # Cache lookup is bypassed for requests whose method is neither of
+    # GET or HEAD.
     - conditions:
       - comparand: req.method
         compare: not-equal

examples/req-disposition/builtin.yaml

@@ -12,8 +12,26 @@ spec:
   services:
     - varnish-ingress
 
+  # req-disposition configures the disposition of client requests, as
+  # an array of objects with the conditions and disposition fields. If
+  # all of the conditions for such an object are true for a request,
+  # then the request is processed further as specified in the
+  # corresponding disposition field. The disposition specifies the
+  # next state of request processing, see:
+  # https://varnish-cache.org/docs/6.1/reference/states.html
+  #
+  # The conditions are evaluated in order, and the first set of
+  # conditions that evaluate to true invoke the corresponding
+  # disposition. If none of the conditions match, the request
+  # proceeds to cache lookup -- return(hash) from vcl_recv.
+  #
+  # This configuration reconstructs vcl_recv from builtin.vcl,
+  # see:
+  # https://github.com/varnishcache/varnish-cache/blob/6.1/bin/varnishd/builtin.vcl
   req-disposition:
 
+    # If the request method is "PRI", then send a synthetic "405
+    # Method Not Allowed" response.
     - conditions:
       - comparand: req.method
         compare: equal
@@ -23,6 +41,8 @@ spec:
         action: synth
         status: 405
 
+    # Reject an HTTP/1.1 request with a synthetic "400 Bad Request"
+    # response if it doesn't have a Host header.
     - conditions:
       - comparand: req.http.Host
         compare: not-exists
@@ -38,6 +58,8 @@ spec:
         action: synth
         status: 400
 
+    # Go to pipe mode for any request whose method is CONNECT, or a
+    # non-standard method.
     - conditions:
       - comparand: req.method
         compare: not-equal
@@ -53,6 +75,10 @@ spec:
       disposition:
         action: pipe
 
+    # Bypass cache lookup if the request method is neither of GET or
+    # HEAD.  Note that we only get here if the previous conditions did
+    # not evaluate to true, so the method must be one of PUT, POST,
+    # TRACE, OPTIONS, DELETE or PATCH.
     - conditions:
       - comparand: req.method
         compare: not-equal
@@ -62,14 +88,21 @@ spec:
       disposition:
         action: pass
 
+    # Bypass cache lookup if the request has a Cookie header.
     - conditions:
       - comparand: req.http.Cookie
         compare: exists
       disposition:
         action: pass
 
+    # Bypass cache lookup if the request has an Authorization header;
+    # that is, if it is sending credentials for basic or proxy
+    # authentication.
     - conditions:
       - comparand: req.http.Authorization
         compare: exists
       disposition:
         action: pass
+
+# If none of the conditions evaluate to true, then the request
+# proceeds to cache lookup.

examples/req-disposition/cacheability.yaml

@@ -14,8 +14,11 @@ spec:
   services:
     - varnish-ingress
 
+  # Configure the disposition of client requests.
   req-disposition:
 
+    # Requests for HTTP/1.1. without a Host header are rejected with
+    # 400 Bad Request, as in builtin.yaml and built-in vcl_recv.
     - conditions:
       - comparand: req.http.Host
         compare: not-exists
@@ -31,6 +34,8 @@ spec:
         action: synth
         status: 400
 
+    # Requests with any non-standard method get a synthetic 405 Method
+    # Not Allowed response.
     - conditions:
       - comparand: req.method
         compare: not-equal
@@ -47,6 +52,8 @@ spec:
         action: synth
         status: 405
 
+    # Cache lookup is bypassed for requests whose method is neither of
+    # GET or HEAD.
     - conditions:
       - comparand: req.method
         compare: not-equal
@@ -56,6 +63,8 @@ spec:
       disposition:
         action: pass
 
+    # Go to cache lookup (hash) if the URL ends in any of the patterns
+    # listed in values.
     - conditions:
       - comparand: req.url
         compare: match
@@ -67,6 +76,8 @@ spec:
       disposition:
         action: hash
 
+    # Bypass cache lookup (pass) if the URL begins with any of the
+    # prefixes listed in values.
     - conditions:
       - comparand: req.url
         compare: prefix

examples/req-disposition/pass-on-session-cookie.yaml

+# Configuration for disposition of client requests that illustrates
+# bypassing cache lookups for specific cookies.
+
+apiVersion: "ingress.varnish-cache.org/v1alpha1"
+kind: VarnishConfig
+metadata:
+  name: pass-on-session-cookie-cfg
+spec:
+  # The services array is required and must have at least one element.
+  # Lists the Service names of Varnish services in the same namespace
+  # to which this config is to be applied.
+  services:
+    - varnish-ingress
+
+  # Configure the disposition of client requests.
+  req-disposition:
+
+    # Requests for HTTP/1.1. without a Host header are rejected with
+    # 400 Bad Request, as in builtin.yaml and built-in vcl_recv.
+    - conditions:
+      - comparand: req.http.Host
+        compare: not-exists
+      - comparand: req.esi_level
+        count: 0
+      - comparand: req.proto
+        compare: prefix
+        values:
+          - HTTP/1.1
+        match-flags:
+          case-insensitive: true
+      disposition:
+        action: synth
+        status: 400
+
+    # Requests with any non-standard method get a synthetic 405 Method
+    # Not Allowed response.
+    - conditions:
+      - comparand: req.method
+        compare: not-equal
+        values:
+          - GET
+          - HEAD
+          - PUT
+          - POST
+          - TRACE
+          - OPTIONS
+          - DELETE
+          - PATCH
+          - CONNECT
+      disposition:
+        action: synth
+        status: 405
+
+    # Cache lookup is bypassed if the Cookie header includes either of
+    # the SESSION or LOGINID cookies, with their values constrained to
+    # specific forms. Cache lookup is permitted otherwise.
+    - conditions:
+      - comparand: req.http.Cookie
+        compare: match
+        values:
+          - \bSESSIONID\s*=\s*[[:xdigit:]]{32}\b
+          - \bLOGIN\s*=\s*\w+\b
+      disposition:
+        action: pass
+
+    # Cache lookup is bypassed for requests whose method is neither of
+    # GET or HEAD.
+    - conditions:
+      - comparand: req.method
+        compare: not-equal
+        values:
+          - GET
+          - HEAD
+      disposition:
+        action: pass

examples/req-disposition/purge-method.yaml

@@ -13,8 +13,11 @@ spec:
   services:
     - varnish-ingress
 
+  # Configure the disposition of client requests.
   req-disposition:
 
+    # Requests for HTTP/1.1. without a Host header are rejected with
+    # 400 Bad Request, as in builtin.yaml and built-in vcl_recv.
     - conditions:
       - comparand: req.http.Host
         compare: not-exists
@@ -30,6 +33,8 @@ spec:
         action: synth
         status: 400
 
+    # Divert to purge when the request method is PURGE.
+    # See: https://varnish-cache.org/docs/6.1/users-guide/purging.html
     - conditions:
       - comparand: req.method
         compare: equal
@@ -38,6 +43,8 @@ spec:
       disposition:
         action: purge
 
+    # Requests with any non-standard method get a synthetic 405 Method
+    # Not Allowed response.
     - conditions:
       - comparand: req.method
         compare: not-equal
@@ -50,10 +57,13 @@ spec:
           - OPTIONS
           - DELETE
           - PATCH
+          - CONNECT
       disposition:
         action: synth
         status: 405
 
+    # Cache lookup is bypassed for requests whose method is neither of
+    # GET or HEAD.
     - conditions:
       - comparand: req.method
         compare: not-equal

examples/req-disposition/url-whitelist.yaml

 # Configuration for disposition of client requests that permits cache
 # lookups for requests with Cookie or Authorization headers, and
-# defines a whitelist for requests based on URL path prefixes.
+# defines white- and blacklists for requests based on URL path prefixes.
 
 apiVersion: "ingress.varnish-cache.org/v1alpha1"
 kind: VarnishConfig
@@ -13,8 +13,11 @@ spec:
   services:
     - varnish-ingress
 
+  # Configure the disposition of client requests.
   req-disposition:
 
+    # Requests for HTTP/1.1. without a Host header are rejected with
+    # 400 Bad Request, as in builtin.yaml and built-in vcl_recv.
     - conditions:
       - comparand: req.http.Host
         compare: not-exists
@@ -30,6 +33,8 @@ spec:
         action: synth
         status: 400
 
+    # Requests with any non-standard method get a synthetic 405 Method
+    # Not Allowed response.
     - conditions:
       - comparand: req.method
         compare: not-equal
@@ -42,25 +47,42 @@ spec:
           - OPTIONS
           - DELETE
           - PATCH
+          - CONNECT
       disposition:
         action: synth
         status: 405
 
+    # A URL whitelist. Requests for URLs that do not have these
+    # prefixes get a synthetic 403 Forbidden response.
     - conditions:
-      - comparand: req.method
-        compare: not-equal
+      - comparand: req.url
+        compare: not-prefix
         values:
-          - GET
-          - HEAD
+          - /tea/sugar/
+          - /coffee/sugar/
       disposition:
-        action: pass
+        action: synth
+        status: 403
 
+    # A URL blacklist. Requests for URLs with these prefixes get a
+    # synthetic 403 Forbidden response.
     - conditions:
       - comparand: req.url
-        compare: not-prefix
+        compare: prefix
         values:
-          - /tea/
-          - /coffee/
+          - /tea/sugar/black/
+          - /coffee/sugar/black/
       disposition:
         action: synth
         status: 403
+
+    # Cache lookup is bypassed for requests whose method is neither of
+    # GET or HEAD.
+    - conditions:
+      - comparand: req.method
+        compare: not-equal
+        values:
+          - GET
+          - HEAD
+      disposition:
+        action: pass