More doc updates.

README.rst

@@ -198,12 +198,12 @@ to match against sets of patterns, consider using the set interface
 of `VMOD re2`_, which provides techniques similar to the present VMOD.
 
 The limited expressiveness of strings to be matched means that this
-VMOD can apply fast algorithms. While regexen and a VMOD like re2 can
-be used to match fixed strings and prefixes, the matching operations
-of VMOD selector are orders of magnitude faster. That in turn
-contributes to scalability by consuming less CPU time for matches. So
-if your use case allows matches against strings without patterns,
-prefer the use of this VMOD.
+VMOD can implement fast algorithms. While regexen and a VMOD like re2
+can be used to match fixed strings and prefixes, the matching
+operations of VMOD selector are orders of magnitude faster. That in
+turn contributes to scalability by consuming less CPU time for
+matches. So if your use case allows matches against strings without
+patterns, prefer the use of this VMOD.
 
 Selecting matched elements of a set
 -----------------------------------
@@ -395,15 +395,13 @@ A regular expression in the ``regex`` parameter is compiled at VCL load
 time. If the compile fails, then the VCL load fails with an error message.
 Regular expressions are evaluated exactly as native regexen in VCL.
 
-``.add()`` fails and invokes VCL failure (see `ERRORS`_) under the
-following conditions:
+``.add()`` invokes VCL failure if it is called in any subroutine
+besides ``vcl_init``. The VCL load fails if:
 
-* ``.add()`` is called any subroutine besides ``vcl_init``.
+* The set has already been compiled.
 
 * The string to be added is NULL.
 
-* The same string is added to the same set more than once.
-
 * A regular expression in the ``regex`` parameter fails to compile.
 
 Example::
@@ -415,6 +413,7 @@ Example::
       myset.add("www.baz.com", string="/baz", backend=baz_backend);
       myset.add("www.quux.com", string="/quux", backend=quux_backend,
       		regex="^/quux/([^/]+)/");
+      myset.compile();
   }
 
 .. _xset.compile():
@@ -422,29 +421,47 @@ Example::
 VOID xset.compile()
 -------------------
 
-Compile the set, to prepare for the ``.match()`` operation.
+Compile the set. This must be done for every set, after all of the
+strings have been added.
+
+``.compile()`` invokes VCL failure if it is called in any subroutine
+besides ``vcl_init``. The VCL load fails if:
+
+* The same string is added to the same set more than once (that string
+  is included in the error message).
+
+* The set contains a string that is a prefix of another string in the
+  same set, but ``allow_overlaps`` was set to ``false`` in the
+  constructor.
+
+* The set has already been compiled.
+
+* The compilation fails for any other reason (probably out of memory).
+
+To re-state the rules about the order of ``.add()`` and ``.compile()``
+more simply: in ``vcl_init``, add all of the strings to the set, then
+compile the set.
+
+If no strings were added to the set before ``.compile()`` is invoked,
+the VCL load will not fail, but all match operations on the set will
+fail. In that case, a warning is emitted to the log with the
+``VCL_Error`` tag. Since that happens outside of any request/response
+transaction, the error message can only be seen when a tool like
+``varniwhlog(1)`` is used with raw grouping (``-g raw``).
 
 .. _xset.create_stats():
 
 VOID xset.create_stats()
 ------------------------
 
-Creates statistics counters for this object that are displayed by
-tools such as ``varnishstat(1)``. See `STATISTICS`_ below for details.
-It should be called in ``vcl_init`` after all strings have been added
-to the set. No statistics are created for a set object if
-``.create_stats()`` is not invoked.
+Create statistics counters for this object that are displayed by tools
+such as ``varnishstat(1)``. See `STATISTICS`_ for details.  It must be
+called in ``vcl_init`` after the set is compiled. No statistics are
+created for a set object if ``.create_stats()`` is not invoked.
 
-Unlike the matching operations, the time needed for this method
-increases as the number of strings in the set increases, since it
-traverses the entire internal data structure. For large sets, the time
-needed for a VCL load can become noticeably longer. If that is a
-problem, consider using this method during development and testing,
-and removing it for production deployments (since the stats values are
-always the same for sets with the same strings).
-
-``.create_stats()`` fails and invokes VCL failure if it is called in
-any VCL subroutine besides ``vcl_init``.
+``.create_stats()`` invokes VCL failure if it is called in any VCL
+subroutine besides ``vcl_init``. The VCL load fails if it is called
+before the set is compiled.
 
 Example::
 
@@ -453,6 +470,7 @@ Example::
       myset.add("foo");
       myset.add("bar");
       myset.add("baz");
+      myset.compile();
       myset.create_stats();
   }
 
@@ -466,18 +484,30 @@ strings in the set. The match is case insensitive if and only if the
 parameter ``case_sensitive`` was set to ``false`` in the set
 constructor (matches are case sensitive by default).
 
-``.match()`` fails and returns ``false`` under the following conditions:
-
-* The string to be matched is NULL (such as an unset header).
+``.match()`` invokes VCL failure if:
 
 * No strings were added to the set.
 
 * There is insufficient workspace for internal operations.
 
-Example::
+If the string to be matched is NULL, for example when an unset header
+is unspecified, then ``.match()`` returns ``false``, and a warning is
+emitted to the log with the ``Notice`` header (see `LOGGING`_). This
+is because a match against an unset header may or may not have been
+intentional.
 
-  if (myset.match(req.http.Host)) {
-     call do_on_match;
+If you need to distinguish whether or not the header exists when using
+``.match()``, you can evaluate the header in boolean context::
+
+  if (!myset.match(req.http.Foo)) {
+      # Either there is no such header in the client request, or
+      # the header does not match the set.
+      # ...
+  }
+
+  if (req.http.Foo && !myset.match(req.http.Foo)) {
+      # The header exists, but does not match the set.
+      # ...
   }
 
 .. _xset.hasprefix():
@@ -965,6 +995,11 @@ a VCL subroutine:
 * If the failure occurs in ``vcl_synth``, then ``vcl_synth`` is
   aborted, and the response line "503 VCL failed" is sent.
 
+LOGGING
+=======
+
+XXX ...
+
 REQUIREMENTS
 ============
 

src/vmod_selector.vcc

@@ -194,12 +194,12 @@ to match against sets of patterns, consider using the set interface
 of `VMOD re2`_, which provides techniques similar to the present VMOD.
 
 The limited expressiveness of strings to be matched means that this
-VMOD can apply fast algorithms. While regexen and a VMOD like re2 can
-be used to match fixed strings and prefixes, the matching operations
-of VMOD selector are orders of magnitude faster. That in turn
-contributes to scalability by consuming less CPU time for matches. So
-if your use case allows matches against strings without patterns,
-prefer the use of this VMOD.
+VMOD can implement fast algorithms. While regexen and a VMOD like re2
+can be used to match fixed strings and prefixes, the matching
+operations of VMOD selector are orders of magnitude faster. That in
+turn contributes to scalability by consuming less CPU time for
+matches. So if your use case allows matches against strings without
+patterns, prefer the use of this VMOD.
 
 Selecting matched elements of a set
 -----------------------------------
@@ -369,15 +369,13 @@ A regular expression in the ``regex`` parameter is compiled at VCL load
 time. If the compile fails, then the VCL load fails with an error message.
 Regular expressions are evaluated exactly as native regexen in VCL.
 
-``.add()`` fails and invokes VCL failure (see `ERRORS`_) under the
-following conditions:
+``.add()`` invokes VCL failure if it is called in any subroutine
+besides ``vcl_init``. The VCL load fails if:
 
-* ``.add()`` is called any subroutine besides ``vcl_init``.
+* The set has already been compiled.
 
 * The string to be added is NULL.
 
-* The same string is added to the same set more than once.
-
 * A regular expression in the ``regex`` parameter fails to compile.
 
 Example::
@@ -389,30 +387,49 @@ Example::
       myset.add("www.baz.com", string="/baz", backend=baz_backend);
       myset.add("www.quux.com", string="/quux", backend=quux_backend,
       		regex="^/quux/([^/]+)/");
+      myset.compile();
   }
 
 $Method VOID .compile()
 
-Compile the set, to prepare for the ``.match()`` operation.
+Compile the set. This must be done for every set, after all of the
+strings have been added.
 
-$Method VOID .create_stats(PRIV_VCL)
+``.compile()`` invokes VCL failure if it is called in any subroutine
+besides ``vcl_init``. The VCL load fails if:
+
+* The same string is added to the same set more than once (that string
+  is included in the error message).
+
+* The set contains a string that is a prefix of another string in the
+  same set, but ``allow_overlaps`` was set to ``false`` in the
+  constructor.
+
+* The set has already been compiled.
 
-Creates statistics counters for this object that are displayed by
-tools such as ``varnishstat(1)``. See `STATISTICS`_ below for details.
-It should be called in ``vcl_init`` after all strings have been added
-to the set. No statistics are created for a set object if
-``.create_stats()`` is not invoked.
+* The compilation fails for any other reason (probably out of memory).
 
-Unlike the matching operations, the time needed for this method
-increases as the number of strings in the set increases, since it
-traverses the entire internal data structure. For large sets, the time
-needed for a VCL load can become noticeably longer. If that is a
-problem, consider using this method during development and testing,
-and removing it for production deployments (since the stats values are
-always the same for sets with the same strings).
+To re-state the rules about the order of ``.add()`` and ``.compile()``
+more simply: in ``vcl_init``, add all of the strings to the set, then
+compile the set.
 
-``.create_stats()`` fails and invokes VCL failure if it is called in
-any VCL subroutine besides ``vcl_init``.
+If no strings were added to the set before ``.compile()`` is invoked,
+the VCL load will not fail, but all match operations on the set will
+fail. In that case, a warning is emitted to the log with the
+``VCL_Error`` tag. Since that happens outside of any request/response
+transaction, the error message can only be seen when a tool like
+``varniwhlog(1)`` is used with raw grouping (``-g raw``).
+
+$Method VOID .create_stats(PRIV_VCL)
+
+Create statistics counters for this object that are displayed by tools
+such as ``varnishstat(1)``. See `STATISTICS`_ for details.  It must be
+called in ``vcl_init`` after the set is compiled. No statistics are
+created for a set object if ``.create_stats()`` is not invoked.
+
+``.create_stats()`` invokes VCL failure if it is called in any VCL
+subroutine besides ``vcl_init``. The VCL load fails if it is called
+before the set is compiled.
 
 Example::
 
@@ -421,6 +438,7 @@ Example::
       myset.add("foo");
       myset.add("bar");
       myset.add("baz");
+      myset.compile();
       myset.create_stats();
   }
 
@@ -431,18 +449,30 @@ strings in the set. The match is case insensitive if and only if the
 parameter ``case_sensitive`` was set to ``false`` in the set
 constructor (matches are case sensitive by default).
 
-``.match()`` fails and returns ``false`` under the following conditions:
-
-* The string to be matched is NULL (such as an unset header).
+``.match()`` invokes VCL failure if:
 
 * No strings were added to the set.
 
 * There is insufficient workspace for internal operations.
 
-Example::
+If the string to be matched is NULL, for example when an unset header
+is unspecified, then ``.match()`` returns ``false``, and a warning is
+emitted to the log with the ``Notice`` header (see `LOGGING`_). This
+is because a match against an unset header may or may not have been
+intentional.
 
-  if (myset.match(req.http.Host)) {
-     call do_on_match;
+If you need to distinguish whether or not the header exists when using
+``.match()``, you can evaluate the header in boolean context::
+
+  if (!myset.match(req.http.Foo)) {
+      # Either there is no such header in the client request, or
+      # the header does not match the set.
+      # ...
+  }
+
+  if (req.http.Foo && !myset.match(req.http.Foo)) {
+      # The header exists, but does not match the set.
+      # ...
   }
 
 $Method BOOL .hasprefix(STRING)
@@ -853,6 +883,11 @@ a VCL subroutine:
 * If the failure occurs in ``vcl_synth``, then ``vcl_synth`` is
   aborted, and the response line "503 VCL failed" is sent.
 
+LOGGING
+=======
+
+XXX ...
+
 REQUIREMENTS
 ============