..
.. NB:  This file is machine generated, DO NOT EDIT!
..
.. Edit ./vdfp_pipe.vcc and run make instead
..

.. role:: ref(emphasis)

=========
vmod_pipe
=========

-----------------------------------------------------------------------------------
Varnish Delivery and Fetch Processors that pipe responses through external commands
-----------------------------------------------------------------------------------

:Manual section: 3

SYNOPSIS
========

::

  import pipe;

  # Delivery processor object
  new <obj> = pipe.vdp(STRING path [, STRING name] [, BYTES bufsz]
                       [, DURATION timeout])
  <obj>.arg(STRING)

  sub vcl_deliver {
	set resp.filters = "<vdp_name>";
  }

  # Fetch processor object
  # XXX: currently not implemented
  new <obj> = pipe.vfp(STRING path)
  <obj>.arg(STRING)

  sub vcl_backend_response {
	set beresp.filters = "<vfp_name>";
  }

  # Version string
  STRING pipe.version()

DESCRIPTION
===========

.. _Varnish: http://www.varnish-cache.org/

The pipe library provides `Varnish`_ delivery and fetch processors
(VDPs and VFPs) that pipe client or backend responses through an
external command. When a processor runs, the Varnish worker process
forks and executes the command. The response is written to the process
standard input stream, and is transformed to the resulting standard
output.

XXX: Currently only the VDP is implemented.

The processors also read the process standard error stream, and write
any stderr output as error messages to the Varnish log.

VDPs and VFPs are technically similar to Varnish Modules (VMODs). In
particular, a processor must be installed against Varnish in the same
way a VMOD is installed; and as with a VMOD, a VCL source must
instruct Varnish to load the processor using the ``import``
statement. But unlike a VMOD, a processor's primary purpose is not to
extend the VCL language. Instead, a VDP/VFP creates filters for
client/backend responses that can be named as strings in the
space-separated lists assigned to the VCL variables ``resp.filters``
and ``beresp.filters``, respectively (see ``vcl(7)``).

For example, this code applies a VDP that sorts the lines in a client
response, and then another one that removes duplicate lines::

  import pipe;

  sub vcl_init {
	new sort = pipe.vdp("/usr/bin/sort");
	new uniq = pipe.vdp("/usr/bin/uniq");
  }

  sub vcl_deliver {
	set resp.filters = "sort uniq";
  }

The processors can be chained with Varnish built-in processors that
perform operations such as edge-side includes (ESI) or compression.
For example, to execute the VDPs shown above after ESI processing::

  set resp.filters = "esi sort uniq";

.. _pipe.vdp():

new xvdp = pipe.vdp(STRING path, STRING name, BYTES bufsz, DURATION timeout)
----------------------------------------------------------------------------

::

   new xvdp = pipe.vdp(
      STRING path,
      STRING name=0,
      BYTES bufsz=0,
      DURATION timeout=60
   )

Define a VDP that transforms client responses by piping them through
the program located at ``path``.

The ``path`` parameter is required, MAY NOT be empty, and MUST specify
the absolute location of the file; the ``PATH`` environment variable
is not used to search for the program. The file at that location MUST
be accessible and executable at VCL initialization time, otherwise the
VCL load fails. This is just a spot check; the command is invoked anew
every time it is specified in ``resp.filters``, and of course the file
may be removed, or its permissions changed, at any time.  In that
case, the VDP will fail at runtime (see `ERRORS`_ below).

The ``path`` parameter should specify the executable's location,
nothing more. Do not, for example, add spaces and command-line
arguments; use the ``.arg()`` method for that (see below). No shell
expansion is applied to the string.

The environment of the varnishd worker process is passed to the
program when it is invoked.

The optional ``name`` parameter specifies a name to be used for the
VDP in ``resp.filters``. If ``name`` is not set, then the VDP is
identified by the symbol name of the object; this is the default. For
example::

  sub vcl_init {
	new sort = pipe.vdp("/usr/bin/sort");
	new uniq = pipe.vdp("/usr/bin/uniq", name="dedup");
  }

  sub vcl_deliver {
	set resp.filters = "sort dedup";
  }

The optional ``bufsz`` parameter sets the size of the buffer used to
read chunks of data from stdout. Set a VCL BYTES value, such as ``8k``
or ``1MB``. If ``bufsz`` is 0b, the buffer size is set to the value of
``PIPE_BUF`` when the pipe library was compiled. ``PIPE_BUF`` is
defined by POSIX to be at least 512 bytes (4KB on Linux). This is the
default. You may be able to determine its value with the command
``getconf PIPE_BUF /``.

.. |fetch_chunksize| replace:: ``fetch_chunksize``
.. _fetch_chunksize: https://varnish-cache.org/docs/trunk/reference/varnishd.html#fetch-chunksize

If ``bufsz`` is too small, then the VDP is less efficient, iterating
through stdout more often than necessary. If ``bufsz`` is too large,
then it wastes memory. For large responses, it may be advantageous to
set ``bufsz`` to the same value as the varnishd parameter
|fetch_chunksize|_ (default 16KB in recent Varnish versions), since
Varnish uses buffers with that size internally. There is probably no
benefit to setting ``bufsz`` larger than ``fetch_chunksize``.

For example::

  sub vcl_init {
	# Set the buffer size to match default fetch_chunksize.
	new sort = pipe.vdp("/usr/bin/sort", bufsz=16k);
  }

.. |first_byte_timeout| replace:: ``first_byte_timeout``
.. _first_byte_timeout: https://varnish-cache.org/docs/trunk/reference/varnishd.html#first-byte-timeout

.. |between_bytes_timeout| replace:: ``between_bytes_timeout``
.. _between_bytes_timeout: https://varnish-cache.org/docs/trunk/reference/varnishd.html#between-bytes-timeout

The optional ``timeout`` parameter sets the timeout waiting for I/O to
become ready at the stdin, stdout and stderr streams. This is a
timeout per iteration, not a timeout for the complete operation. The
default is 60s (matching the default values of the varnishd parameters
|first_byte_timeout|_ and |between_bytes_timeout|_ in recent Varnish
versions).

If the timeout expires before I/O is ready at any of the three
streams, then the VDP fails (see `ERRORS`_ below). Long timeouts will
block the VDP if the external process is hanging or excessively slow.

If ``timeout`` is negative (for example ``timeout=-1s``), then the VDP
waits indefinitely for I/O. If ``timeout`` is 0s, then the VDP does
not block waiting for I/O, and repeats the iteration immediately if
none of the streams are ready. This is likely to cause inefficient
busy-wait loops.

A complete example::

  sub vcl_init {
	new uniq = pipe.vdp(path="/usr/bin/uniq",
	                    name="dedup", bufsz=16k, timeout=10s);
  }

.. _xvdp.arg():

VOID xvdp.arg(STRING)
---------------------

The ``.arg()`` method sets a command-line argument to be used with a
program.

Restricted to: ``vcl_init``, ``vcl_deliver``.

Invocations of ``.arg()`` in ``vcl_init`` set arguments to be used
globally in the VCL instance. Invocations in ``vcl_deliver`` set
arguments to be used for a single client response, overriding any
arguments that may have been set in ``vcl_init`` For example, you can
call ``.arg()`` in ``vcl_deliver`` if the arguments to be used are not
known until runtime.

The parameter in ``.arg()`` MAY be empty (if you need the empty string
as a CLI argument), but MAY NOT be NULL (for example, it may not be
set from an unset header).

The CLI arguments for a command are ordered as the order in which
``.arg()`` is invoked in VCL. For example::

  sub vcl_init {
	# Use the tr command, by default to uppercase all ASCII
	# alphabet characters in client responses.
	new tr = pipe.vdp("/usr/bin/tr");
	tr.arg("a-z");
	tr.arg("A-Z");
  }

  sub vcl_deliver {
	# If these two headers are set, use their values as arguments
	# for tr for the current response. Otherwise, the args set in
	# vcl_init are used.
	if (resp.http.X-TR-Set1 && resp.http.X-TR-Set2) {
		tr.arg(resp.http.X-TR-Set1);
		tr.arg(resp.http.X-TR-Set2);
	}
	set resp.filters = "tr";
  }

You can leave ``.arg()`` out of ``vcl_init``, and only call it in
``vcl_deliver``, if the values of the arguments are only known at
runtime.

Do not use shell-style escaping or quoting in ``.arg()``. The
arguments should be written exactly as the program will use them::

  # This shell command performs the ROT47 transformation.
  $ tr '\!-~' 'P-~\!-O'

  # In .arg(), write the arguments without quoting or escapes.
  sub vcl_init {
	new rot47 = pipe.vdp("/usr/bin/tr");
	rot47.arg("!-~");
	rot47.arg("P-~!-Q");
  }

The example shown above, which chains VDPs for ``sort`` and ``uniq``,
can be implemented more efficiently by using the ``-u`` argument with
``sort``, so that ``uniq`` is unnecessary::

  sub vcl_init {
	new sort = pipe.vdp("/usr/bin/sort");
	sort.arg("-u");
  }

.. _xvdp.setenv():

VOID xvdp.setenv(STRING var, STRING value, BOOL overwrite)
----------------------------------------------------------

::

      VOID xvdp.setenv(STRING var, STRING value, BOOL overwrite=1)

Set the environment variable ``var`` to ``value`` in the invoked
process.

Restricted to: ``vcl_init``, ``vcl_deliver``.

Settings in ``vcl_init`` are global for the VCL instance, while
settings in ``vcl_deliver`` are valid for a single client response
(for use cases where the environment settings are not known until
runtime).

Currently, any invocation of ``.setenv()`` in ``vcl_deliver`` cancels
all environment settings specified in ``vcl_init`` for the current
client response. For example, if you set a value for the variable
``FOO`` in ``vcl_init`` and ``BAR`` in ``vcl_deliver``, then ``FOO`` is
not set, unless you also set ``FOO`` in ``vcl_deliver`` (if necessary
to the same value).

The variable name ``var`` MAY NOT be empty or NULL, and MAY NOT
contain the equals sign (``=``). ``value`` MAY be empty (to set the
empty string as the value), but MAY NOT be NULL. A string is NULL, for
example, if you specify an unset header.

If the optional parameter ``overwrite`` is ``true``, then if ``var``
is already set in the process environment (for example, due to
inheritance from the varnishd worker process, or after a previous
invocation of ``.setenv()`` for the same variable), then its value is
changed to ``value``. If ``overwrite`` is ``false``, any previous
value is left unchanged (this is not an error).  By default,
``overwrite`` is ``true``.

Environment variables are set in the same order as ``.setenv()``
invocations in VCL. So if the same variable is set by ``.setenv()``
more than once, then the value in the last invocation is set, unless
``overwrite`` is false in that invocation.

For example::

  sub vcl_init {
	# Global environment settings, unless overridden in vcl_deliver.
	new app = pipe.vdp("/usr/bin/myapp");
	app.setenv("FOO", "bar");
	app.setenv("BAZ", "quux");
	app.setenv("NOTHING", "");
	# Set this value for LANG unless it is already set due to
	# inheritance from varnishd.
	app.setenv("LANG", "de_DE.UTF8", overwrite=false);
  }

  sub vcl_deliver {
	# For this URL, set different values in the environment.
	if (req.url == "/bazooka") {
		# NOTHING and LANG not set for this response.
		app.setenv("FOO", "fighter");
		app.setenv("BAZ", "ooka");
	}
	set resp.filters = "app";
  }

.. _pipe.version():

STRING version()
----------------

Return the version string for this library.

Example::

  std.log("Using pipe version: " + pipe.version());

ERRORS
======

XXX ...

REQUIREMENTS
============

The Varnish master branch is currently required.

INSTALLATION
============

See `INSTALL.rst <INSTALL.rst>`_ in the source repository.

SEE ALSO
========

* source repository website: https://code.uplex.de/uplex-varnish/libvdfp-pipe
* Varnish: http://www.varnish-cache.org/
* varnishd(1): http://varnish-cache.org/docs/trunk/reference/varnishd.html
* vcl(7): http://varnish-cache.org/docs/trunk/reference/vcl.html

COPYRIGHT
=========

::

  Copyright (c) 2019 UPLEX Nils Goroll Systemoptimierung
  All rights reserved
 
  Author: Geoffrey Simmons <geoffrey.simmons@uplex.de>
 
  See LICENSE