Commit f89ff340 authored by Geoff Simmons's avatar Geoff Simmons

Start docs.

parent 86f39c3f
......@@ -25,16 +25,18 @@ SYNOPSIS
import pipe;
# Delivery processor object
new <obj> = pipe.vdp(XXX)
<obj>.args(XXX)
new <obj> = pipe.vdp(STRING path [, STRING name] [, BYTES bufsz]
[, DURATION timeout])
<obj>.arg(STRING)
sub vcl_deliver {
set resp.filters = "<vdp_name>";
}
# Fetch processor object
new <obj> = pipe.vfp(XXX)
<obj>.args(XXX)
# XXX: currently not implemented
new <obj> = pipe.vfp(STRING path)
<obj>.arg(STRING)
sub vcl_backend_response {
set beresp.filters = "<vfp_name>";
......@@ -49,10 +51,46 @@ DESCRIPTION
.. _Varnish: http://www.varnish-cache.org/
The pipe library provides `Varnish`_ delivery and fetch processors
(VDPs and VFPs) that pipe client and backend responses through
external commands.
(VDPs and VFPs) that pipe client or backend responses through an
external command. When a processor runs, the Varnish worker process
forks and executes the command. The response is written to the process
standard input stream, and is transformed to the resulting standard
output.
XXX: Currently only the VDP is implemented.
The processors also read the process standard error stream, and write
any stderr output as error messages to the Varnish log.
VDPs and VFPs are technically similar to Varnish Modules (VMODs). In
particular, a processor must be installed against Varnish in the same
way a VMOD is installed; and as with a VMOD, a VCL source must
instruct Varnish to load the processor using the ``import``
statement. But unlike a VMOD, a processor's primary purpose is not to
extend the VCL language. Instead, a VDP/VFP creates filters for
client/backend responses that can be named as strings in the
space-separated lists assigned to the VCL variables ``resp.filters``
and ``beresp.filters``, respectively (see ``vcl(7)``).
For example, this code applies a VDP that sorts the lines in a client
response, and then another one that removes duplicate lines::
XXX ...
import pipe;
sub vcl_init {
new sort = pipe.vdp("/usr/bin/sort");
new uniq = pipe.vdp("/usr/bin/uniq");
}
sub vcl_deliver {
set resp.filters = "sort uniq";
}
The processors can be chained with Varnish built-in processors that
perform operations such as edge-side includes (ESI) or compression.
For example, to execute the VDPs shown above after ESI processing::
set resp.filters = "esi sort uniq";
.. _pipe.vdp():
......@@ -68,14 +106,160 @@ new xvdp = pipe.vdp(STRING path, STRING name, BYTES bufsz, DURATION timeout)
DURATION timeout=60
)
XXX ...
Define a VDP that transforms client responses by piping them through
the program located at ``path``. The constructor may only be called in
``vcl_init``.
The ``path`` parameter is required, MAY NOT be empty, and MUST specify
the absolute location of the file; the ``PATH`` environment variable
is not used to search for the program. The file at that location MUST
be accessible and executable at VCL initialization time, otherwise the
VCL load fails. This is just a spot check; the command is invoked anew
every time it is specified in ``resp.filters``, and of course the file
may be removed, or its permissions changed, at any time. In that
case, the VDP will fail at runtime (see `ERRORS`_ below).
The ``path`` parameter should specify the executable's location,
nothing more. Do not, for example, add spaces and command-line
arguments; use the ``.arg()`` method for that (see below). No shell
expansion is applied to the string.
The optional ``name`` parameter specifies a name to be used for the
VDP in ``resp.filters``. If ``name`` is not set, then the VDP is
identified by the symbol name of the object; this is the default. For
example::
sub vcl_init {
new sort = pipe.vdp("/usr/bin/sort");
new uniq = pipe.vdp("/usr/bin/uniq", name="dedup");
}
sub vcl_deliver {
set resp.filters = "sort dedup";
}
The optional ``bufsz`` parameter sets the size of the buffer used to
read chunks of data from stdout. Set a VCL BYTES value, such as ``8k``
or ``1MB``. If ``bufsz`` is 0b, the buffer size is set to the value of
``PIPE_BUF`` when the pipe library was compiled. ``PIPE_BUF`` is
defined by POSIX to be at least 512 bytes (4KB on Linux). This is the
default. You may be able to determine its value with the command
``getconf PIPE_BUF /``.
.. |fetch_chunksize| replace:: ``fetch_chunksize``
.. _fetch_chunksize: https://varnish-cache.org/docs/trunk/reference/varnishd.html#fetch-chunksize
If ``bufsz`` is too small, then the VDP is less efficient, iterating
through stdout more often than necessary. If ``bufsz`` is too large,
then it wastes memory. For large responses, it may be advantageous to
set ``bufsz`` to the same value as the varnishd parameter
|fetch_chunksize|_ (default 16KB in recent Varnish versions), since
Varnish uses buffers with that size internally. There is probably no
benefit to setting ``bufsz`` larger than ``fetch_chunksize``.
For example::
sub vcl_init {
# Set the buffer size to match default fetch_chunksize.
new sort = pipe.vdp("/usr/bin/sort", bufsz=16k);
}
.. |first_byte_timeout| replace:: ``first_byte_timeout``
.. _first_byte_timeout: https://varnish-cache.org/docs/trunk/reference/varnishd.html#first-byte-timeout
.. |between_bytes_timeout| replace:: ``between_bytes_timeout``
.. _between_bytes_timeout: https://varnish-cache.org/docs/trunk/reference/varnishd.html#between-bytes-timeout
The optional ``timeout`` parameter sets the timeout waiting for I/O to
become ready at the stdin, stdout and stderr streams. This is a
timeout per iteration, not a timeout for the complete operation. The
default is 60s (matching the default values of the varnishd parameters
|first_byte_timeout|_ and |between_bytes_timeout|_ in recent Varnish
versions).
If the timeout expires before I/O is ready at any of the three
streams, then the VDP fails (see `ERRORS`_ below). Long timeouts will
block the VDP if the external process is hanging or excessively slow.
If ``timeout`` is negative (for example ``timeout=-1s``), then the VDP
waits indefinitely for I/O. If ``timeout`` is 0s, then the VDP does
not block waiting for I/O, and repeats the iteration immediately if
none of the streams are ready. This is likely to cause inefficient
busy-wait loops.
A complete example::
sub vcl_init {
new uniq = pipe.vdp(path="/usr/bin/uniq",
name="dedup", bufsz=16k, timeout=10s);
}
.. _xvdp.arg():
VOID xvdp.arg(STRING)
---------------------
XXX ...
The ``.arg()`` method sets a command-line argument to be used with a
program. The method MAY NOT be called in any VCL subroutine besides
``vcl_init`` or ``vcl_deliver``. Invocations of ``.arg()`` in
``vcl_init`` set arguments to be used globally in the VCL
instance. Invocations in ``vcl_deliver`` set arguments to be used for
a single client response, overriding any arguments that may have been
set in ``vcl_init`` For example, you can call ``.arg()`` in
``vcl_deliver`` if the arguments to be used are not known until
runtime.
The parameter in ``.arg()`` MAY be empty (if you need the empty string
as a CLI argument), but MAY NOT be NULL (for example, it may not be
set from an unset header).
The CLI arguments for a command are ordered as the order in which
``.arg()`` is invoked in VCL. For example::
sub vcl_init {
# Use the tr command, by default to uppercase all ASCII
# alphabet characters in client responses.
new tr = pipe.vdp("/usr/bin/tr");
tr.arg("a-z");
tr.arg("A-Z");
}
sub vcl_deliver {
# If these two headers are set, use their values as arguments
# for tr for the current response. Otherwise, the args set in
# vcl_init are used.
if (resp.http.X-TR-Set1 && resp.http.X-TR-Set2) {
tr.arg(resp.http.X-TR-Set1);
tr.arg(resp.http.X-TR-Set2);
}
set resp.filters = "tr";
}
You can leave ``.arg()`` out of ``vcl_init``, and only call it in
``vcl_deliver``, if the values of the arguments are only known at
runtime.
Do not use shell-style escaping or quoting in ``.arg()``. The
arguments should be written exactly as the program will use them::
# This shell command performs the ROT47 transformation.
$ tr '\!-~' 'P-~\!-O'
# In .arg(), write the arguments without quoting or escapes.
sub vcl_init {
new rot47 = pipe.vdp("/usr/bin/tr");
rot47.arg("!-~");
rot47.arg("P-~!-Q");
}
The example shown above, which chains VDPs for ``sort`` and ``uniq``,
can be implemented more efficiently by using the ``-u`` argument with
``sort``, so that ``uniq`` is unnecessary::
sub vcl_init {
new sort = pipe.vdp("/usr/bin/sort");
sort.arg("-u");
}
.. _pipe.version():
......
......@@ -21,16 +21,18 @@ SYNOPSIS
import pipe;
# Delivery processor object
new <obj> = pipe.vdp(XXX)
<obj>.args(XXX)
new <obj> = pipe.vdp(STRING path [, STRING name] [, BYTES bufsz]
[, DURATION timeout])
<obj>.arg(STRING)
sub vcl_deliver {
set resp.filters = "<vdp_name>";
}
# Fetch processor object
new <obj> = pipe.vfp(XXX)
<obj>.args(XXX)
# XXX: currently not implemented
new <obj> = pipe.vfp(STRING path)
<obj>.arg(STRING)
sub vcl_backend_response {
set beresp.filters = "<vfp_name>";
......@@ -45,18 +47,200 @@ DESCRIPTION
.. _Varnish: http://www.varnish-cache.org/
The pipe library provides `Varnish`_ delivery and fetch processors
(VDPs and VFPs) that pipe client and backend responses through
external commands.
(VDPs and VFPs) that pipe client or backend responses through an
external command. When a processor runs, the Varnish worker process
forks and executes the command. The response is written to the process
standard input stream, and is transformed to the resulting standard
output.
XXX: Currently only the VDP is implemented.
The processors also read the process standard error stream, and write
any stderr output as error messages to the Varnish log.
VDPs and VFPs are technically similar to Varnish Modules (VMODs). In
particular, a processor must be installed against Varnish in the same
way a VMOD is installed; and as with a VMOD, a VCL source must
instruct Varnish to load the processor using the ``import``
statement. But unlike a VMOD, a processor's primary purpose is not to
extend the VCL language. Instead, a VDP/VFP creates filters for
client/backend responses that can be named as strings in the
space-separated lists assigned to the VCL variables ``resp.filters``
and ``beresp.filters``, respectively (see ``vcl(7)``).
For example, this code applies a VDP that sorts the lines in a client
response, and then another one that removes duplicate lines::
XXX ...
import pipe;
sub vcl_init {
new sort = pipe.vdp("/usr/bin/sort");
new uniq = pipe.vdp("/usr/bin/uniq");
}
sub vcl_deliver {
set resp.filters = "sort uniq";
}
The processors can be chained with Varnish built-in processors that
perform operations such as edge-side includes (ESI) or compression.
For example, to execute the VDPs shown above after ESI processing::
set resp.filters = "esi sort uniq";
$Object vdp(STRING path, STRING name=0, BYTES bufsz=0, DURATION timeout=60)
XXX ...
Define a VDP that transforms client responses by piping them through
the program located at ``path``. The constructor may only be called in
``vcl_init``.
The ``path`` parameter is required, MAY NOT be empty, and MUST specify
the absolute location of the file; the ``PATH`` environment variable
is not used to search for the program. The file at that location MUST
be accessible and executable at VCL initialization time, otherwise the
VCL load fails. This is just a spot check; the command is invoked anew
every time it is specified in ``resp.filters``, and of course the file
may be removed, or its permissions changed, at any time. In that
case, the VDP will fail at runtime (see `ERRORS`_ below).
The ``path`` parameter should specify the executable's location,
nothing more. Do not, for example, add spaces and command-line
arguments; use the ``.arg()`` method for that (see below). No shell
expansion is applied to the string.
The optional ``name`` parameter specifies a name to be used for the
VDP in ``resp.filters``. If ``name`` is not set, then the VDP is
identified by the symbol name of the object; this is the default. For
example::
sub vcl_init {
new sort = pipe.vdp("/usr/bin/sort");
new uniq = pipe.vdp("/usr/bin/uniq", name="dedup");
}
sub vcl_deliver {
set resp.filters = "sort dedup";
}
The optional ``bufsz`` parameter sets the size of the buffer used to
read chunks of data from stdout. Set a VCL BYTES value, such as ``8k``
or ``1MB``. If ``bufsz`` is 0b, the buffer size is set to the value of
``PIPE_BUF`` when the pipe library was compiled. ``PIPE_BUF`` is
defined by POSIX to be at least 512 bytes (4KB on Linux). This is the
default. You may be able to determine its value with the command
``getconf PIPE_BUF /``.
.. |fetch_chunksize| replace:: ``fetch_chunksize``
.. _fetch_chunksize: https://varnish-cache.org/docs/trunk/reference/varnishd.html#fetch-chunksize
If ``bufsz`` is too small, then the VDP is less efficient, iterating
through stdout more often than necessary. If ``bufsz`` is too large,
then it wastes memory. For large responses, it may be advantageous to
set ``bufsz`` to the same value as the varnishd parameter
|fetch_chunksize|_ (default 16KB in recent Varnish versions), since
Varnish uses buffers with that size internally. There is probably no
benefit to setting ``bufsz`` larger than ``fetch_chunksize``.
For example::
sub vcl_init {
# Set the buffer size to match default fetch_chunksize.
new sort = pipe.vdp("/usr/bin/sort", bufsz=16k);
}
.. |first_byte_timeout| replace:: ``first_byte_timeout``
.. _first_byte_timeout: https://varnish-cache.org/docs/trunk/reference/varnishd.html#first-byte-timeout
.. |between_bytes_timeout| replace:: ``between_bytes_timeout``
.. _between_bytes_timeout: https://varnish-cache.org/docs/trunk/reference/varnishd.html#between-bytes-timeout
The optional ``timeout`` parameter sets the timeout waiting for I/O to
become ready at the stdin, stdout and stderr streams. This is a
timeout per iteration, not a timeout for the complete operation. The
default is 60s (matching the default values of the varnishd parameters
|first_byte_timeout|_ and |between_bytes_timeout|_ in recent Varnish
versions).
If the timeout expires before I/O is ready at any of the three
streams, then the VDP fails (see `ERRORS`_ below). Long timeouts will
block the VDP if the external process is hanging or excessively slow.
If ``timeout`` is negative (for example ``timeout=-1s``), then the VDP
waits indefinitely for I/O. If ``timeout`` is 0s, then the VDP does
not block waiting for I/O, and repeats the iteration immediately if
none of the streams are ready. This is likely to cause inefficient
busy-wait loops.
A complete example::
sub vcl_init {
new uniq = pipe.vdp(path="/usr/bin/uniq",
name="dedup", bufsz=16k, timeout=10s);
}
$Method VOID .arg(STRING)
XXX ...
The ``.arg()`` method sets a command-line argument to be used with a
program. The method MAY NOT be called in any VCL subroutine besides
``vcl_init`` or ``vcl_deliver``. Invocations of ``.arg()`` in
``vcl_init`` set arguments to be used globally in the VCL
instance. Invocations in ``vcl_deliver`` set arguments to be used for
a single client response, overriding any arguments that may have been
set in ``vcl_init`` For example, you can call ``.arg()`` in
``vcl_deliver`` if the arguments to be used are not known until
runtime.
The parameter in ``.arg()`` MAY be empty (if you need the empty string
as a CLI argument), but MAY NOT be NULL (for example, it may not be
set from an unset header).
The CLI arguments for a command are ordered as the order in which
``.arg()`` is invoked in VCL. For example::
sub vcl_init {
# Use the tr command, by default to uppercase all ASCII
# alphabet characters in client responses.
new tr = pipe.vdp("/usr/bin/tr");
tr.arg("a-z");
tr.arg("A-Z");
}
sub vcl_deliver {
# If these two headers are set, use their values as arguments
# for tr for the current response. Otherwise, the args set in
# vcl_init are used.
if (resp.http.X-TR-Set1 && resp.http.X-TR-Set2) {
tr.arg(resp.http.X-TR-Set1);
tr.arg(resp.http.X-TR-Set2);
}
set resp.filters = "tr";
}
You can leave ``.arg()`` out of ``vcl_init``, and only call it in
``vcl_deliver``, if the values of the arguments are only known at
runtime.
Do not use shell-style escaping or quoting in ``.arg()``. The
arguments should be written exactly as the program will use them::
# This shell command performs the ROT47 transformation.
$ tr '\!-~' 'P-~\!-O'
# In .arg(), write the arguments without quoting or escapes.
sub vcl_init {
new rot47 = pipe.vdp("/usr/bin/tr");
rot47.arg("!-~");
rot47.arg("P-~!-Q");
}
The example shown above, which chains VDPs for ``sort`` and ``uniq``,
can be implemented more efficiently by using the ``-u`` argument with
``sort``, so that ``uniq`` is unnecessary::
sub vcl_init {
new sort = pipe.vdp("/usr/bin/sort");
sort.arg("-u");
}
$Function STRING version()
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment