whats-new: changes

doc/sphinx/whats-new/changes-trunk.rst

@@ -25,54 +25,468 @@ merged, may be found in the `change log`_.
 varnishd
 ========
 
+Arguments
+~~~~~~~~~
+
+* ``varnishd`` now supports the ``-b None`` argument to start with
+  only the builtin VCL and no backend at all.
+
 Parameters
 ~~~~~~~~~~
 
-**XXX changes in -p parameters**
+* The ``validate_headers`` parameter has been added to control
+  `header validation <whatsnew_changes_CURRENT_header_validation_>`_.
+
+* The ``ban_cutoff`` parameter now refers to the overall length of the
+  ban list, including completed bans, where before only non-completed
+  ("active") bans were counted towards ``ban_cutoff``.
+
+* The ``vary_notice`` parameter has been added to control the
+  threshold for the new `Vary Notice
+  <whatsnew_changes_CURRENT_vary_notice_>`_.
+
+``feature`` Flags
+~~~~~~~~~~~~~~~~~
+
+* The ``busy_stats_rate`` feature flag has been added to ensure
+  statistics updates (as configured using the ``thread_stats_rate``
+  parameter) even on a fully loaded system, which would otherwise
+  delay statistics updates in order to reduce lock contention.
+
+.. _whatsnew_changes_CURRENT_accounting:
+
+Accounting
+~~~~~~~~~~
+
+Body bytes accounting has been fixed to always represent the number of
+bodybytes moved on the wire, exclusive of protocol-specific overhead
+like HTTP/1 chunked encoding or HTTP/2 framing.
+
+This change affects counters like
+
+- ``MAIN.s_req_bodybytes``,
+
+- ``MAIN.s_resp_bodybytes``,
+
+- ``VBE.*.*.bereq_bodybytes`` and
+
+- ``VBE.*.*.beresp_bodybytes``
+
+as well as the VSL records
+
+- ``ReqAcct``,
+
+- ``PipeAcct`` and
 
-Other changes in varnishd
-~~~~~~~~~~~~~~~~~~~~~~~~~
+- ``BereqAcct``.
+
+.. _whatsnew_changes_CURRENT_sc_close:
+
+Session Close Reasons
+~~~~~~~~~~~~~~~~~~~~~
+
+The connection close reason has been fixed to properly report
+``SC_RESP_CLOSE`` / ``resp_close`` where previously only
+``SC_REQ_CLOSE`` / ``req_close`` was reported.
+
+For failing PROXY connections, ``SessClose`` now provides more
+detailed information on the cause of the failure.
+
+The session close reason logging/statistics for HTTP/2 connections
+have been improved.
+
+.. _whatsnew_changes_CURRENT_vary_notice:
+
+Vary Notice
+~~~~~~~~~~~
+
+A log (VSL) ``Notice`` record is now emitted whenever more than
+``vary_notice`` variants are encountered in the cache for a specific
+hash. The new ``vary_notice`` parameter defaults to 10.
 
 Changes to VCL
 ==============
 
+.. _whatsnew_changes_CURRENT_header_validation:
+
+Header Validation
+~~~~~~~~~~~~~~~~~
+
+Unless the new ``validate_headers`` feature is disabled, all newly set
+headers are now validated to contain only characters allowed by
+RFC7230. A (runtime) VCL failure is triggered if not.
+
 VCL variables
 ~~~~~~~~~~~~~
 
-**XXX new, deprecated or removed variables, or changed semantics**
+* The ``client.identity`` variable is now accessible on the backend
+  side.
+
+* The variables ``bereq.is_hitpass`` and ``bereq.is_hitmiss`` have
+  been added to the backend side matching ``req.is_hitpass`` and
+  ``req.is_hitmiss`` on the client side.
+
+* The ``bereq.xid`` variable is now also available in ``vcl_pipe {}``
+
+* The ``resp.proto`` variable is now read-only as it should have been
+  for long.
 
 Other changes to VCL
 ~~~~~~~~~~~~~~~~~~~~
 
+* Long strings in VCL can now also be denoted using ``""" ... """`` in
+  addition to the existing ``{" ... "}``.
+
+* The ``ban()`` builtin is now deprecated and should be replaced with
+  `std.ban() <whatsnew_changes_CURRENT_ban_>`_.
+
+* Trying to use ``std.rollback()`` from ``vcl_pipe`` now results in
+  VCL failure.
+
+* The modulus operator ``%`` has been added to VCL.
+
+* ``return(retry)`` from ``vcl_backend_error {}`` now correctly resets
+  ``beresp.status`` and ``beresp.reason``.
+
+* The builtin VCL has been reworked: VCL code has been split into
+  small subroutines, which custom VCL can prepend custom code to.
+
+  This allows for better integration of custom VCL and the built-in
+  VCL and better reuse.
+
 VMODs
 =====
 
-**XXX changes in the bundled VMODs**
+``directors.shard()``
+~~~~~~~~~~~~~~~~~~~~~
+
+* The shard director now supports reconfiguration (adding/removing
+  backends) of several instances without any special ordering
+  requirement.
+
+* Calling the shard director ``.reconfigure()`` method is now
+  optional. If not called explicitly, any shard director backend
+  changes are applied at the end of the current task.
+
+* Shard director ``Error`` log messages with ``(notice)`` have been
+  turned into ``Notice`` log messages.
+
+* All shard ``Error`` and ``Notice`` messages now use the unified
+  prefix ``vmod_directors: shard %s``.
+
+``std.set_ip_tos()``
+~~~~~~~~~~~~~~~~~~~~
+
+The ``set_ip_tos()`` function from the bundled ``std`` vmod now sets
+the IPv6 Taffic Class (TCLASS) when used on an IPv6 connection.
+
+.. _whatsnew_changes_CURRENT_ban:
+
+``std.ban()`` and ``std.ban_error()``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The ``std.ban()`` and ``std.ban_error()`` functions have been added to
+the ``std`` vmod, allowing VCL to check for ban errors. A typical
+usage pattern with the new interface is::
+
+  if (std.ban(...)) {
+    return(synth(200, "Ban added"));
+  } else {
+    return(synth(400, std.ban_error()));
+  }
+
+``cookie`` functions
+~~~~~~~~~~~~~~~~~~~~
+
+The ``filter_re``, ``keep_re`` and ``get_re`` functions from the
+bundled ``cookie`` vmod have been changed to take the ``VCL_REGEX``
+type. This implies that their regular expression arguments now need to
+be literal, whereas before they could be taken from some other
+variable or function returning ``VCL_STRING``.
+
+Note that these functions never actually handled *dynamic* regexen,
+the string passed with the first call was compiled to a regex, which
+was then used for the lifetime of the respective VCL.
+
 
 varnishlog
 ==========
 
-**XXX changes concerning varnishlog(1) and/or vsl(7)**
+* See `Accounting <whatsnew_changes_CURRENT_accounting_>`_ for changes
+  to accounting-related VSL records.
+
+* See `Session Close Reasons <whatsnew_changes_CURRENT_sc_close_>`_
+  for a change affecting ``SessClose``.
+
+* Three new ``Timestamp`` VSL records have been added to backend
+  request processing:
+
+  - The ``Process`` timestamp after ``return(deliver)`` or
+    ``return(pass(x))`` from ``vcl_backend_response``,
+
+  - the ``Fetch`` timestamp before a backend connection is requested
+    and
+
+  - the ``Connected`` timestamp when a connection to a regular backend
+    (VBE) is established.
+
+* The ``FetchError`` log message ``Timed out reusing backend
+  connection`` has been renamed to ``first byte timeout (reused
+  connection)`` to clarify that it is emit for effectively the same
+  reason as ``first byte timeout``.
+
+* ``ExpKill`` log (VSL) records are now masked by default. See the
+  ``vsl_mask`` parameter.
+
+* Comparisons of numbers in VSL queries have been improved to match
+  better the behavior which is likely expected by users who have not
+  read the documentation in all detail.
+
+* See `Vary Notice <whatsnew_changes_CURRENT_vary_notice_>`_ for
+  information on a newly added ``Notice`` log (VSL) record.
+
+varnishncsa
+===========
+
+* The ``%{X}T`` format has been added to ``varnishncsa``, which
+  generalizes ``%D`` and ``%T``, but also support milliseconds
+  (``ms``) output.
+
+* The ``varnishncsa`` ``-E`` argument to show ESI requests has been
+  changed to imply ``-c`` (client mode).
+
 
 varnishadm
 ==========
 
-**XXX changes concerning varnishadm(1) and/or varnish-cli(7)**
+* The ``vcl.discard`` CLI command can now be used to discard more than
+  one VCL with a single command, which succeeds only if all given VCLs
+  could be discarded (atomic behavior).
+
+* The ``vcl.discard`` CLI command now supports glob patterns for vcl names.
+
+* The ``vcl.deps`` CLI command has been added to output dependencies
+  between VCLs (because of labels and ``return(vcl)`` statements).
+
+* ``varnishadm`` now has the ``-p`` option to disable readline support
+  for use in scripts and as a generic CLI connector.
 
 varnishstat
 ===========
 
-**XXX changes concerning varnishstat(1) and/or varnish-counters(7)**
+* See `Accounting <whatsnew_changes_CURRENT_accounting_>`_ for changes
+  to accounting-related counters.
+
+* See `Session Close Reasons <whatsnew_changes_CURRENT_sc_close_>`_
+  for a change affecting ``MAIN.sc_*`` counters.
+
+* The ``MAIN.esi_req`` counter has been added as a statistic of the
+  number of ESI sub requests created.
+
+* The ``MAIN.s_bgfetch`` counter has been added as a statistic on the
+  number of background fetches issued.
+
+.. _whatsnew_changes_CURRENT_varnishstat_raw:
+
+* ``varnishstat`` now avoids display errors of gauges which previously
+  could underflow to negative values, being displayed as extremely
+  high positive values.
+
+  The ``-r`` option and the ``r`` key binding have been added to
+  return to the previous behavior. When raw mode is active in
+  ``varnishstat`` interactive (curses) mode, the word ``RAW`` is
+  displayed at the right hand side in the lower status line.
 
 varnishtest
 ===========
 
-**XXX changes concerning varnishtest(1) and/or vtc(7)**
+Various improvements have been made to the ``varnishtest`` facility:
+
+- the ``loop`` keyword now works everywhere
+
+- HTTP/2 logging has been improved
+
+- Default HTTP/2 parameters have been tweaked
+
+- Varnish listen address information is now available by default in
+  the macros ``${vNAME_addr}``, ``${vNAME_port}`` and
+  ``${vNAME_sock}``. Macros by the names ``${vNAME_SOCKET_*}`` contain
+  the address information for each listen socket as created with the
+  ``-a`` argument to ``varnishd``.
+
+- Synchronization points for counters (VSCs) have been added as
+  ``varnish vNAME -expect PATTERN OP PATTERN``
+
+- varnishtest now also works with IPv6 setups
+
+- ``feature ipqv4`` and ``feature ipv6`` can be used to control
+  execution of test cases which require one or the other protocol.
+
+- haproxy arguments can now be externally provided through the
+  ``HAPROXY_ARGS`` variable.
+
+- logexpect now supports alternatives with the ``expect ? ...`` syntax
+  and negative matches with the ``fail add ...`` and ``fail clear``
+  syntax.
+
+- The overall logexpect match expectation can now be inverted using
+  the ``-err`` argument.
+
+- Numeric comparisons for HTTP headers have been added: ``-lt``,
+  ``-le``, ``-eq``, ``-ne``, ``-ge``, ``-gt``
+
+- ``rxdata -some`` has been fixed.
+
+Other Changes to Varnish Utilities
+==================================
+
+All varnish tools using the VUT library utilities for argument
+processing now support the ``--optstring`` argument to return a string
+suitable for use with ``getopts`` from shell scripts.
+
+
+Developer: Changes for VMOD authors
+===================================
+
+VMOD/VCL interface
+~~~~~~~~~~~~~~~~~~
+
+* The ``VCL_REGEX`` data type is now supported for VMODs, allowing
+  them to use regular expression literals checked and compiled by the
+  VCL compiler infrastructure.
+
+  Consequently, the ``VRT_re_init()`` and ``VRT_re_fini()`` functions
+  have been removed, because they are not required and their use was
+  probably wrong anyway.
+
+* The ``VCL_SUB`` data type is now supported for VMODs to save
+  references to subroutines to be called later using
+  ``VRT_call()``. Calls from a wrong context (e.g. calling a
+  subroutine accessing ``req`` from the backend side) and recursive
+  calls fail the VCL.
+
+  See `VMOD - Varnish Modules`_ in the Reference Manual.
+
+.. _VMOD - Varnish Modules: https://varnish-cache.org/docs/trunk/reference/vmod.html
+
+  VMOD functions can also return the ``VCL_SUB`` data type for calls
+  from VCL as in ``call vmod.returning_sub();``.
+
+* ``VRT_check_call()`` can be used to check if a ``VRT_call()`` would
+  succeed in order to avoid the potential VCL failure in case it would
+  not.
+
+  It returns ``NULL`` if ``VRT_call()`` would make the call or an
+  error string why not.
+
+* ``VRT_handled()`` has been added, which is now to be used instead of
+  access to the ``handling`` member of ``VRT_CTX``.
+
+* ``vmodtool.py`` has been improved to simplify Makefiles when many
+  VMODs are built in a single directory.
 
-Changes for developers and VMOD authors
-=======================================
+General API
+~~~~~~~~~~~
+
+* ``VRT_ValidHdr()`` has been added for vmods to conduct the same
+  check as the `whatsnew_changes_CURRENT_header_validation`_ feature,
+  for example when headers are set by vmods using the ``cache_http.c``
+  Functions like ``http_ForceHeader()`` from untrusted input.
+
+* Client and backend finite state machine internals (``enum req_step``
+  and ``enum fetch_step``) have been removed from ``cache.h``.
+
+* The ``verrno.h`` header file has been removed and merged into
+  ``vas.h``
+
+* The ``pdiff()`` function declaration has been moved from ``cache.h``
+  to ``vas.h``.
+
+VSA
+~~~
+
+* The ``VSA_getsockname()`` and ``VSA_getpeername()`` functions have
+  been added to get address information of file descriptors.
+
+Private Pointers
+~~~~~~~~~~~~~~~~
+
+* The interface for private pointers in VMODs has been changed:
+
+  - The ``free`` pointer in ``struct vmod_priv`` has been replaced
+    with a pointer to ``struct vmod_priv_methods``, to where the
+    pointer to the former free callback has been moved as the ``fini``
+    member.
+
+  - The former free callback type has been renamed from
+    ``vmod_priv_free_f`` to ``vmod_priv_fini_f`` and as gained a
+    ``VRT_CTX`` argument
+
+* The ``VRT_priv_task_get()`` and ``VRT_priv_top_get()`` functions
+  have been added to VRT to allow vmods to retrieve existing
+  ``PRIV_TASK`` / ``PRIV_TOP`` private pointers without creating any.
+
+Backends
+~~~~~~~~
+
+* The VRT backend interface has been changed:
+
+  - ``struct vrt_endpoint`` has been added describing a UDS or TCP
+    endpoint for a backend to connect to.
+
+    Endpoints also support a preamble to be sent with every new
+    connection.
+
+  - This structure needs to be passed via the ``endpoint`` member of
+    ``struct vrt_backend`` when creating backends with
+    ``VRT_new_backend()`` or ``VRT_new_backend_clustered()``.
+
+* ``VRT_Endpoint_Clone()`` has been added to facilitate working with
+  endpoints.
+
+Filters (VDP/VFP)
+~~~~~~~~~~~~~~~~~
+
+* Many filter (VDP/VFP) related signatures have been changed:
+
+  - ``vdp_init_f``
+
+  - ``vdp_fini_f``
+
+  - ``vdp_bytes_f``
+
+  - ``VDP_bytes()``
+
+  as well as ``struct vdp_entry`` and ``struct vdp_ctx``
+
+  ``VFP_Push()`` and ``VDP_Push()`` are no longer intended for VMOD
+  use and have been removed from the API.
+
+* The VDP code is now more strict about ``VDP_END``, which must be
+  sent down the filter chain at most once. Care should be taken to
+  send ``VDP_END`` together with the last payload bytes whenever
+  possible.
+
+Stevedore API
+~~~~~~~~~~~~~
+
+* The stevedore API has been changed:
+
+  - ``OBJ_ITER_FINAL`` has been renamed to ``OBJ_ITER_END``
+
+  - ``ObjExtend()`` signature has been changed to also cover the
+    ``ObjTrimStore()`` use case and
+
+  - ``ObjTrimStore()`` has been removed.
+
+Developer: Changes for Authors of Varnish Utilities
+===================================================
+
+libvarnishapi
+~~~~~~~~~~~~~
 
-**XXX changes concerning VRT, the public APIs, source code organization,
-builds etc.**
+* The ``VSC_IsRaw()`` function has been added to ``libvarnishapi`` to
+  query if a gauge is being returned raw or adjusted (see
+  `varnishstat -r option <whatsnew_changes_CURRENT_varnishstat_raw_>`_).
 
 *eof*