README.rst 12.1 KB
Newer Older
1 2 3
..
.. NB:  This file is machine generated, DO NOT EDIT!
..
Geoff Simmons's avatar
Geoff Simmons committed
4
.. Edit ./vdfp_pipe.vcc and run make instead
5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27
..

.. role:: ref(emphasis)

=========
vmod_pipe
=========

-----------------------------------------------------------------------------------
Varnish Delivery and Fetch Processors that pipe responses through external commands
-----------------------------------------------------------------------------------

:Manual section: 3


SYNOPSIS
========

::

  import pipe;

  # Delivery processor object
Geoff Simmons's avatar
Geoff Simmons committed
28 29 30
  new <obj> = pipe.vdp(STRING path [, STRING name] [, BYTES bufsz]
                       [, DURATION timeout])
  <obj>.arg(STRING)
31 32 33 34 35 36

  sub vcl_deliver {
	set resp.filters = "<vdp_name>";
  }

  # Fetch processor object
Geoff Simmons's avatar
Geoff Simmons committed
37 38 39
  # XXX: currently not implemented
  new <obj> = pipe.vfp(STRING path)
  <obj>.arg(STRING)
40 41 42 43 44 45 46 47 48 49 50 51 52 53

  sub vcl_backend_response {
	set beresp.filters = "<vfp_name>";
  }

  # Version string
  STRING pipe.version()

DESCRIPTION
===========

.. _Varnish: http://www.varnish-cache.org/

The pipe library provides `Varnish`_ delivery and fetch processors
Geoff Simmons's avatar
Geoff Simmons committed
54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76
(VDPs and VFPs) that pipe client or backend responses through an
external command. When a processor runs, the Varnish worker process
forks and executes the command. The response is written to the process
standard input stream, and is transformed to the resulting standard
output.

XXX: Currently only the VDP is implemented.

The processors also read the process standard error stream, and write
any stderr output as error messages to the Varnish log.

VDPs and VFPs are technically similar to Varnish Modules (VMODs). In
particular, a processor must be installed against Varnish in the same
way a VMOD is installed; and as with a VMOD, a VCL source must
instruct Varnish to load the processor using the ``import``
statement. But unlike a VMOD, a processor's primary purpose is not to
extend the VCL language. Instead, a VDP/VFP creates filters for
client/backend responses that can be named as strings in the
space-separated lists assigned to the VCL variables ``resp.filters``
and ``beresp.filters``, respectively (see ``vcl(7)``).

For example, this code applies a VDP that sorts the lines in a client
response, and then another one that removes duplicate lines::
77

Geoff Simmons's avatar
Geoff Simmons committed
78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93
  import pipe;

  sub vcl_init {
	new sort = pipe.vdp("/usr/bin/sort");
	new uniq = pipe.vdp("/usr/bin/uniq");
  }

  sub vcl_deliver {
	set resp.filters = "sort uniq";
  }

The processors can be chained with Varnish built-in processors that
perform operations such as edge-side includes (ESI) or compression.
For example, to execute the VDPs shown above after ESI processing::

  set resp.filters = "esi sort uniq";
94 95 96

.. _pipe.vdp():

97 98
new xvdp = pipe.vdp(STRING path, STRING name, BYTES bufsz, DURATION timeout)
----------------------------------------------------------------------------
99 100 101

::

102 103 104 105 106 107
   new xvdp = pipe.vdp(
      STRING path,
      STRING name=0,
      BYTES bufsz=0,
      DURATION timeout=60
   )
108

Geoff Simmons's avatar
Geoff Simmons committed
109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126
Define a VDP that transforms client responses by piping them through
the program located at ``path``. The constructor may only be called in
``vcl_init``.

The ``path`` parameter is required, MAY NOT be empty, and MUST specify
the absolute location of the file; the ``PATH`` environment variable
is not used to search for the program. The file at that location MUST
be accessible and executable at VCL initialization time, otherwise the
VCL load fails. This is just a spot check; the command is invoked anew
every time it is specified in ``resp.filters``, and of course the file
may be removed, or its permissions changed, at any time.  In that
case, the VDP will fail at runtime (see `ERRORS`_ below).

The ``path`` parameter should specify the executable's location,
nothing more. Do not, for example, add spaces and command-line
arguments; use the ``.arg()`` method for that (see below). No shell
expansion is applied to the string.

127 128 129
The environment of the varnishd worker process is passed to the
program when it is invoked.

Geoff Simmons's avatar
Geoff Simmons committed
130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198
The optional ``name`` parameter specifies a name to be used for the
VDP in ``resp.filters``. If ``name`` is not set, then the VDP is
identified by the symbol name of the object; this is the default. For
example::

  sub vcl_init {
	new sort = pipe.vdp("/usr/bin/sort");
	new uniq = pipe.vdp("/usr/bin/uniq", name="dedup");
  }

  sub vcl_deliver {
	set resp.filters = "sort dedup";
  }

The optional ``bufsz`` parameter sets the size of the buffer used to
read chunks of data from stdout. Set a VCL BYTES value, such as ``8k``
or ``1MB``. If ``bufsz`` is 0b, the buffer size is set to the value of
``PIPE_BUF`` when the pipe library was compiled. ``PIPE_BUF`` is
defined by POSIX to be at least 512 bytes (4KB on Linux). This is the
default. You may be able to determine its value with the command
``getconf PIPE_BUF /``.

.. |fetch_chunksize| replace:: ``fetch_chunksize``
.. _fetch_chunksize: https://varnish-cache.org/docs/trunk/reference/varnishd.html#fetch-chunksize

If ``bufsz`` is too small, then the VDP is less efficient, iterating
through stdout more often than necessary. If ``bufsz`` is too large,
then it wastes memory. For large responses, it may be advantageous to
set ``bufsz`` to the same value as the varnishd parameter
|fetch_chunksize|_ (default 16KB in recent Varnish versions), since
Varnish uses buffers with that size internally. There is probably no
benefit to setting ``bufsz`` larger than ``fetch_chunksize``.

For example::

  sub vcl_init {
	# Set the buffer size to match default fetch_chunksize.
	new sort = pipe.vdp("/usr/bin/sort", bufsz=16k);
  }

.. |first_byte_timeout| replace:: ``first_byte_timeout``
.. _first_byte_timeout: https://varnish-cache.org/docs/trunk/reference/varnishd.html#first-byte-timeout

.. |between_bytes_timeout| replace:: ``between_bytes_timeout``
.. _between_bytes_timeout: https://varnish-cache.org/docs/trunk/reference/varnishd.html#between-bytes-timeout

The optional ``timeout`` parameter sets the timeout waiting for I/O to
become ready at the stdin, stdout and stderr streams. This is a
timeout per iteration, not a timeout for the complete operation. The
default is 60s (matching the default values of the varnishd parameters
|first_byte_timeout|_ and |between_bytes_timeout|_ in recent Varnish
versions).

If the timeout expires before I/O is ready at any of the three
streams, then the VDP fails (see `ERRORS`_ below). Long timeouts will
block the VDP if the external process is hanging or excessively slow.

If ``timeout`` is negative (for example ``timeout=-1s``), then the VDP
waits indefinitely for I/O. If ``timeout`` is 0s, then the VDP does
not block waiting for I/O, and repeats the iteration immediately if
none of the streams are ready. This is likely to cause inefficient
busy-wait loops.

A complete example::

  sub vcl_init {
	new uniq = pipe.vdp(path="/usr/bin/uniq",
	                    name="dedup", bufsz=16k, timeout=10s);
  }
199

200 201 202 203 204
.. _xvdp.arg():

VOID xvdp.arg(STRING)
---------------------

Geoff Simmons's avatar
Geoff Simmons committed
205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265
The ``.arg()`` method sets a command-line argument to be used with a
program. The method MAY NOT be called in any VCL subroutine besides
``vcl_init`` or ``vcl_deliver``. Invocations of ``.arg()`` in
``vcl_init`` set arguments to be used globally in the VCL
instance. Invocations in ``vcl_deliver`` set arguments to be used for
a single client response, overriding any arguments that may have been
set in ``vcl_init`` For example, you can call ``.arg()`` in
``vcl_deliver`` if the arguments to be used are not known until
runtime.

The parameter in ``.arg()`` MAY be empty (if you need the empty string
as a CLI argument), but MAY NOT be NULL (for example, it may not be
set from an unset header).

The CLI arguments for a command are ordered as the order in which
``.arg()`` is invoked in VCL. For example::

  sub vcl_init {
	# Use the tr command, by default to uppercase all ASCII
	# alphabet characters in client responses.
	new tr = pipe.vdp("/usr/bin/tr");
	tr.arg("a-z");
	tr.arg("A-Z");
  }

  sub vcl_deliver {
	# If these two headers are set, use their values as arguments
	# for tr for the current response. Otherwise, the args set in
	# vcl_init are used.
	if (resp.http.X-TR-Set1 && resp.http.X-TR-Set2) {
		tr.arg(resp.http.X-TR-Set1);
		tr.arg(resp.http.X-TR-Set2);
	}
	set resp.filters = "tr";
  }  

You can leave ``.arg()`` out of ``vcl_init``, and only call it in
``vcl_deliver``, if the values of the arguments are only known at
runtime.

Do not use shell-style escaping or quoting in ``.arg()``. The
arguments should be written exactly as the program will use them::

  # This shell command performs the ROT47 transformation.
  $ tr '\!-~' 'P-~\!-O'

  # In .arg(), write the arguments without quoting or escapes.
  sub vcl_init {
	new rot47 = pipe.vdp("/usr/bin/tr");
	rot47.arg("!-~");
	rot47.arg("P-~!-Q");
  }

The example shown above, which chains VDPs for ``sort`` and ``uniq``,
can be implemented more efficiently by using the ``-u`` argument with
``sort``, so that ``uniq`` is unnecessary::

  sub vcl_init {
	new sort = pipe.vdp("/usr/bin/sort");
	sort.arg("-u");
  }
266

Geoff Simmons's avatar
Geoff Simmons committed
267 268 269 270 271 272 273 274 275
.. _xvdp.setenv():

VOID xvdp.setenv(STRING var, STRING value, BOOL overwrite)
----------------------------------------------------------

::

      VOID xvdp.setenv(STRING var, STRING value, BOOL overwrite=1)

Geoff Simmons's avatar
Geoff Simmons committed
276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300
Set the environment variable ``var`` to ``value`` in the invoked
process.

Like ``.arg()``, ``.setenv()`` MAY NOT be called in any VCL subroutine
besides ``vcl_init`` or ``vcl_deliver``. Settings in ``vcl_init`` are
global for the VCL instance, while settings in ``vcl_deliver`` are
valid for a single client response (for use cases where the
environment settings are not known until runtime).

Currently, any invocation of ``.setenv()`` in ``vcl_deliver`` cancels
all environment settings specified in ``vcl_init`` for the current
client response. For example, if you set a value for the variable
``FOO`` in ``vcl_init`` and ``BAR`` in ``vcl_deliver``, then ``FOO`` is
not set, unless you also set ``FOO`` in ``vcl_deliver`` (if necessary
to the same value).

The variable name ``var`` MAY NOT be empty or NULL, and MAY NOT
contain the equals sign (``=``). ``value`` MAY be empty (to set the
empty string as the value), but MAY NOT be NULL. A string is NULL, for
example, if you specify an unset header.

If the optional parameter ``overwrite`` is ``true``, then if ``var``
is already set in the process environment (for example, due to
inheritance from the varnishd worker process, or after a previous
invocation of ``.setenv()`` for the same variable), then its value is
Geoff Simmons's avatar
Geoff Simmons committed
301
changed to ``value``. If ``overwrite`` is ``false``, any previous
Geoff Simmons's avatar
Geoff Simmons committed
302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331
value is left unchanged (this is not an error).  By default,
``overwrite`` is ``true``.

Environment variables are set in the same order as ``.setenv()``
invocations in VCL. So if the same variable is set by ``.setenv()``
more than once, then the value in the last invocation is set, unless
``overwrite`` is false in that invocation.

For example::

  sub vcl_init {
	# Global environment settings, unless overridden in vcl_deliver.
	new app = pipe.vdp("/usr/bin/myapp");
	app.setenv("FOO", "bar");
	app.setenv("BAZ", "quux");
	app.setenv("NOTHING", "");
	# Set this value for LANG unless it is already set due to
	# inheritance from varnishd.
	app.setenv("LANG", "de_DE.UTF8", overwrite=false);
  }

  sub vcl_deliver {
	# For this URL, set different values in the environment.
	if (req.url == "/bazooka") {
		# NOTHING and LANG not set for this response.
		app.setenv("FOO", "fighter");
		app.setenv("BAZ", "ooka");
	}
	set resp.filters = "app";
  }
Geoff Simmons's avatar
Geoff Simmons committed
332

333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383
.. _pipe.version():

STRING version()
----------------

Return the version string for this library.

Example::

  std.log("Using pipe version: " + pipe.version());

ERRORS
======

XXX ...

REQUIREMENTS
============

The Varnish master branch is currently required.

INSTALLATION
============

See `INSTALL.rst <INSTALL.rst>`_ in the source repository.

LIMITATIONS
===========

XXX ...

SEE ALSO
========

* source repository website: https://code.uplex.de/uplex-varnish/libvdfp-pipe
* Varnish: http://www.varnish-cache.org/
* varnishd(1): http://varnish-cache.org/docs/trunk/reference/varnishd.html
* vcl(7): http://varnish-cache.org/docs/trunk/reference/vcl.html

COPYRIGHT
=========

::

  Copyright (c) 2019 UPLEX Nils Goroll Systemoptimierung
  All rights reserved
 
  Author: Geoffrey Simmons <geoffrey.simmons@uplex.de>
 
  See LICENSE