Skip to content

L
libvfp-brotli
Commit 5682ccfc authored by Geoff Simmons's avatar Geoff Simmons
Browse files
Options

Start docs -- SYNOPSIS, DESCRIPTION, and the API docs.

parent 047d3195
Hide whitespace changes
Inline Side-by-side
Showing with 558 additions and 24 deletions
README.rst
View file @ 5682ccfc
......@@ -42,11 +42,11 @@ SYNOPSIS
# Create a brotli compression filter with custom parameters.
new <obj> = brotli.encoder(STRING name, BYTES buffer, INT quality,
INT lgwin)
BOOL large_win, INT lgwin, ENUM mode)
VOID <obj>.create_stats()
# Create a brotli decompression filter with custom parameters.
new <obj> = brotli.decoder(STRING name, BYTES buffer, BOOL lgwin)
new <obj> = brotli.decoder(STRING name, BYTES buffer, BOOL large_win)
VOID <obj>.create_stats()
# VFP version
......@@ -62,9 +62,142 @@ DESCRIPTION
===========
Varnish Fetch Processor (VFP) to support the brotli compresion
algorithm for responses fetched from backends.
algorithm for responses fetched from backends. This VFP integrates
Varnish with the `Google brotli library`_.
The Brotli Compressed Data Format is specified in `RFC 7932`_. Details
of the compression algorithm are beyond the scope of this manual; see
the RFC and library documentation for more information.
A VFP is technically similar to a Varnish Module (VMOD). In
particular, it must be installed against Varnish in the same way a
VMOD is installed, and like a VMOD, a VCL source must instruct Varnish
to load the VFP using the ``import`` statement. But unlike a VMOD, a
VFP's primary purpose is not to extend the VCL language; instead, a
VFP creates filters for backend fetches that can be named as strings
in the space-separated list assigned to the VCL variable
``beresp.filters`` (see ``vcl(7)``).
VFP brotli always creates two filters named ``"br"`` and ``"unbr"``,
for compression and decompression, respectively, with default
parameters for the compression algorithm::
XXX ...
import brotli;
# Use the built-in "br" filter for brotli compression with default
# parameters, if the Accept-Encoding request header indicates that
# brotli compression is accepted.
sub vcl_backend_response {
if (bereq.http.Accept-Encoding ~ "\bbr\b") {
set beresp.filters = "br";
}
}
# Use the built-in "unbr" filter for brotli decompression with
# default parameters, if the Content-Encoding response header
# indicates that the response is compressed using brotli.
sub vcl_backend_response {
if (beresp.http.Content-Encoding == "br") {
set beresp.filters = "unbr";
}
}
Note that the content encoding type ``br`` has been standardized to
indicate brotli compression, for use in headers such as
``Content-Encoding`` and ``Accept-Encoding``.
Note also that ``beresp.filters`` may only be written or read in the
VCL subroutine ``vcl_backend_response``.
When a brotli filter appears in ``beresp.filters``, it is applied to
the incoming backend response body. The resulting response body is
passed to the client, and if the response is cached, then the body is
stored in the cache as it results from the filtering.
To use non-default values for the compression algorithm, create an
``encoder`` or ``decoder`` object as specified below. The string
passed in the ``name`` parameter of the object constructor can be used
in a ``beresp.filters`` list, as with ``"br"`` and
``"unbr"``::
import brotli;
sub vcl_init {
# Create a compression filter with a reduced quality level,
# for faster processing, but larger compressed responses
# (the default quality level is 11).
new brQ10 = brotli.encoder("brQ10", quality=10);
# Create a decompression filter that uses a 1MiB buffer
# (the default buffer size is 32KiB).
new unbr1M = brotli.decoder("unbr1M", buffer=1m);
}
sub vcl_backend_response {
# Use the custom compression filter defined above.
if (bereq.http.Accept-Encoding ~ "\bbr\b") {
set beresp.filters = "brQ10";
}
# Use the custom decompression filter.
if (beresp.http.Content-Encoding == "br") {
set beresp.filters = "unbr1M";
}
}
Parameter settings for the compression algorithm represent various
ways to affect the trade-off between speed of processing and the rate
of compression (how much compression reduces the size of an object).
The VFP always creates counter statistics, observable with a tool like
``varnishstat(1)``, for the standard ``"br"`` and ``"unbr"``
filters. You may also optionally create statistics for custom filters
created with the ``encoder`` and ``decoder`` constructors. See
`STATISTICS`_ below for details.
Compression and HTTP
--------------------
The brotli VFP interacts with the HTTP protocol (headers and response
codes) in much the same way that Varnish does with its built-in gzip
support:
* Compression filters (the built-in ``"br"`` filter, or a custom
filter created from the ``encoder`` constructor) are not executed
for a backend response that already has a ``Content-Encoding``
header.
* Decompression filters (built-in ``"unbr"`` or a custom ``decoder``
filter) are not executed unless the backend response has the
``Content-Encoding`` header set to ``br``.
* When a compression filter is applied, the VFP sets the
``Content-Encoding`` response header to ``br``; this header value
may appear in a client response, and it may be cached with the
response. The value ``Accept-Encoding`` is also added to the
``Vary`` response header.
* When a decompression filter is applied, the ``Content-Encoding``
response header is removed.
* Any ``Content-Length`` response header fetched from the backend is
removed -- Varnish may set a new ``Content-Length`` header in a
client response, set to the changed size of the response body. When
streaming is enabled, Varnish sends the client response for a fetch
with chunked encoding (and hence with no ``Content-Length`` header).
* If the backend response has an ``ETag`` header, then the ``Etag``
value is "weakened" (prefixed with ``W/``), for weak validation
according to `RFC 7232`_ section 2.1. This is because the response
after de-/compression is not byte-identical with the fetched
response.
* The filters ignore partial responses to range requests (status 206
for "Partial Content").
Note that (unlike gzip) brotli compression does not work together with
Edge Side Includes (ESI). See `LIMITATIONS`_ below for a discussion of
this issue, and of possible solutions with VCL.
.. _vmod_brotli.encoder:
......@@ -82,14 +215,100 @@ new xencoder = brotli.encoder(STRING name, BYTES bufffer, INT quality, BOOL larg
ENUM {GENERIC, TEXT, FONT} mode=GENERIC
)
XXX ...
Create a compression filter named ``name`` with custom parameters for
the brotli algorithm. The string given in ``name`` may then be used in
``beresp.filters``.
The default values for the parameters correspond to the defaults used
in the standard compression filter (so if you create a filter with all
defaults, it is functionally identical to the ``"br"`` filter).
The ``name`` MAY NOT be ``"br"`` or ``"unbr"``, or the same as any of
the standard filters built into Varnish: ``"esi"``, ``"esi_gzip"``,
``"gzip"``, ``"gunzip"`` or ``"testgunzip"``. The results are
undefined (and almost certainly not what you want) if you use a name
that is also used by another third-party VFP.
The parameters are:
* ``buffer`` (default 32KiB): Like the ``varnishd`` parameter
``gzip_buffer``, this is the size of the temporary internal buffer
used for compression. As with ``gzip_buffer``, a buffer size that is
too small results in more overhead, and if it is too large, then it
probably wastes memory.
* ``quality`` (default 11): sets a compression level, where 0
represents the fastest execution, and 11 represents the highest
compression rate (smallest compressed result). ``quality`` MUST be
>= 0 and <= 11.
* ``large_win`` (default ``false``): if ``true``, use Large Window
Brotli, which may yield better results for response bodies larger
than 16 MiB. The Large Window algorithm differs from the standard
algorithm; under certain circumstances, it may be necessary to use
the Large Window setting for both compression and decompression.
* ``lgwin`` (default 22): set the window size for the LZ77 algorithm.
The window size is (2 ^ ``lgwin`` - 16), and larger window sizes may
result in better compression rates. ``lgwin`` MUST be >= 10 and <=
24.
* ``mode`` (default ``GENERIC``): provide a hint about the expected
contents of the response body; the compression may benefit from
optimizations based on assumptions about the content. Possible
values are:
* ``GENERIC``: no assumptions are made about the content
* ``TEXT``: for UTF-8 formatted text
* ``FONT``: for the WOFF 2.0 font format (the original application
for brotli)
For more details about the parameters and their effects, see `RFC
7932`_ and the library documentation.
Example::
import brotli;
sub vcl_init {
# Create a compression filter with a 1MiB buffer, optimized
# for WOFF.
new br_woff = brotli.encoder("br_woff", buffer=1m, mode=WOFF);
}
sub vcl_backend_response {
# Use the custom filter for the WOFF media type.
if (beresp.http.Content-Type ~ "\bfont/woff2?\b") {
set beresp.filters = "br_woff";
}
}
.. _vmod_brotli.encoder.create_stats:
VOID xencoder.create_stats()
----------------------------
XXX ...
Create statistics, observable with a tool like ``varnishstat(1)``, for
the custom compression filter. These are the same as the counters
created for the standard ``"br"`` filter. See `STATISTICS`_ below for
details.
The ``.create_stats()`` method MAY NOT be called in any VCL subroutine
besides ``vcl_init``. If it is called in another subroutine, then VCL
failure is invoked -- ordinarily, the request fails with a "503 VCL
failed" response.
Example::
import brotli;
sub vcl_init {
# Create a custom compression filter with stats.
new mybr = brotli.encoder("mybr", buffer=64k);
mybr.create_stats();
}
.. _vmod_brotli.decoder:
......@@ -104,21 +323,61 @@ new xdecoder = brotli.decoder(STRING name, BYTES buffer, BOOL large_win)
BOOL large_win=0
)
XXX ...
Create a decompression filter named ``name`` with custom parameters,
suitable for use in ``beresp.filters``.
As with the ``encoder`` object, the default values for the parameters
correspond to the defaults used in the standard ``"unbr"``
decompression filter.
Also as with ``"encoder``, the ``name`` MAY NOT be ``"br"`` or
``"unbr"``, or the same as any of Varnish's standard filters. It
SHOULD NOT be the same as a name used by another third-party VFP.
The parameters are:
* ``buffer`` (default 32KiB): like the ``buffer`` parameter for the
``encoder`` object, this is the size of the temporary internal
buffer used for decompression.
* ``large_win`` (default ``false``): if ``true``, use Large Window
Brotli. As noted above, it may be necessary to use the Large Window
setting for both compression and decompression.
Example::
import brotli;
sub vcl_init {
# Create a decompression filter with a small buffer size.
new unbr_small = brotli.decoder("unbr_small", buffer=1k);
}
sub vcl_backend_response {
# Use the custom filter when the URL begins with "/tiny"
if (bereq.url ~ "/tiny" && beresp.http.Content-Encoding ~ "br") {
set beresp.filters = "unbr_tiny";
}
}
.. _vmod_brotli.decoder.create_stats:
VOID xdecoder.create_stats()
----------------------------
XXX ...
Create statistics for the custom decompression filter, like the
``.create_stats()`` method for ``encoder``. See `STATISTICS`_ below
for details.
As with ``encoder.create_stats()``, this method MAY NOT be called in
any VCL subroutine besides ``vcl_init``, otherwise VCL failure is invoked.
.. _vmod_brotli.encoder_version:
STRING encoder_version()
------------------------
Return the version string for the brotli encoder.
Return the version string for the brotli encoder library.
Example::
......@@ -129,7 +388,7 @@ Example::
STRING decoder_version()
------------------------
Return the version string for the brotli decoder.
Return the version string for the brotli decoder library.
Example::
......@@ -178,8 +437,10 @@ SEE ALSO
* vcl(7)
* varnishstat(1)
* VMOD source repository: https://code.uplex.de/uplex-varnish/libvfp-brotli
* brotli repository: https://github.com/google/brotli
* RFC 7932: https://tools.ietf.org/html/rfc7932
* brotli library repository: https://github.com/google/brotli
* RFC 7932 (brotli specification): https://tools.ietf.org/html/rfc7932
* RFC 7232 (HTTP conditional requests, with the specification for weak
validation): https://tools.ietf.org/html/rfc7232
COPYRIGHT
=========
......@@ -193,3 +454,9 @@ COPYRIGHT
See LICENSE
.. _Google brotli library: https://github.com/google/brotli
.. _RFC 7932: https://tools.ietf.org/html/rfc7932
.. _RFC 7232: https://tools.ietf.org/html/rfc7232
src/vfp_brotli.vcc
View file @ 5682ccfc
......@@ -7,6 +7,12 @@
# See LICENSE
#
.. _Google brotli library: https://github.com/google/brotli
.. _RFC 7932: https://tools.ietf.org/html/rfc7932
.. _RFC 7232: https://tools.ietf.org/html/rfc7232
$Module brotli 3 "Varnish Fetch Processor for brotli de-/compression"
$ABI vrt
......@@ -38,11 +44,11 @@ SYNOPSIS
# Create a brotli compression filter with custom parameters.
new <obj> = brotli.encoder(STRING name, BYTES buffer, INT quality,
INT lgwin)
BOOL large_win, INT lgwin, ENUM mode)
VOID <obj>.create_stats()
# Create a brotli decompression filter with custom parameters.
new <obj> = brotli.decoder(STRING name, BYTES buffer, BOOL lgwin)
new <obj> = brotli.decoder(STRING name, BYTES buffer, BOOL large_win)
VOID <obj>.create_stats()
# VFP version
......@@ -58,31 +64,290 @@ DESCRIPTION
===========
Varnish Fetch Processor (VFP) to support the brotli compresion
algorithm for responses fetched from backends.
algorithm for responses fetched from backends. This VFP integrates
Varnish with the `Google brotli library`_.
The Brotli Compressed Data Format is specified in `RFC 7932`_. Details
of the compression algorithm are beyond the scope of this manual; see
the RFC and library documentation for more information.
A VFP is technically similar to a Varnish Module (VMOD). In
particular, it must be installed against Varnish in the same way a
VMOD is installed, and like a VMOD, a VCL source must instruct Varnish
to load the VFP using the ``import`` statement. But unlike a VMOD, a
VFP's primary purpose is not to extend the VCL language; instead, a
VFP creates filters for backend fetches that can be named as strings
in the space-separated list assigned to the VCL variable
``beresp.filters`` (see ``vcl(7)``).
VFP brotli always creates two filters named ``"br"`` and ``"unbr"``,
for compression and decompression, respectively, with default
parameters for the compression algorithm::
XXX ...
import brotli;
# Use the built-in "br" filter for brotli compression with default
# parameters, if the Accept-Encoding request header indicates that
# brotli compression is accepted.
sub vcl_backend_response {
if (bereq.http.Accept-Encoding ~ "\bbr\b") {
set beresp.filters = "br";
}
}
# Use the built-in "unbr" filter for brotli decompression with
# default parameters, if the Content-Encoding response header
# indicates that the response is compressed using brotli.
sub vcl_backend_response {
if (beresp.http.Content-Encoding == "br") {
set beresp.filters = "unbr";
}
}
Note that the content encoding type ``br`` has been standardized to
indicate brotli compression, for use in headers such as
``Content-Encoding`` and ``Accept-Encoding``.
Note also that ``beresp.filters`` may only be written or read in the
VCL subroutine ``vcl_backend_response``.
When a brotli filter appears in ``beresp.filters``, it is applied to
the incoming backend response body. The resulting response body is
passed to the client, and if the response is cached, then the body is
stored in the cache as it results from the filtering.
To use non-default values for the compression algorithm, create an
``encoder`` or ``decoder`` object as specified below. The string
passed in the ``name`` parameter of the object constructor can be used
in a ``beresp.filters`` list, as with ``"br"`` and
``"unbr"``::
import brotli;
sub vcl_init {
# Create a compression filter with a reduced quality level,
# for faster processing, but larger compressed responses
# (the default quality level is 11).
new brQ10 = brotli.encoder("brQ10", quality=10);
# Create a decompression filter that uses a 1MiB buffer
# (the default buffer size is 32KiB).
new unbr1M = brotli.decoder("unbr1M", buffer=1m);
}
sub vcl_backend_response {
# Use the custom compression filter defined above.
if (bereq.http.Accept-Encoding ~ "\bbr\b") {
set beresp.filters = "brQ10";
}
# Use the custom decompression filter.
if (beresp.http.Content-Encoding == "br") {
set beresp.filters = "unbr1M";
}
}
Parameter settings for the compression algorithm represent various
ways to affect the trade-off between speed of processing and the rate
of compression (how much compression reduces the size of an object).
The VFP always creates counter statistics, observable with a tool like
``varnishstat(1)``, for the standard ``"br"`` and ``"unbr"``
filters. You may also optionally create statistics for custom filters
created with the ``encoder`` and ``decoder`` constructors. See
`STATISTICS`_ below for details.
Compression and HTTP
--------------------
The brotli VFP interacts with the HTTP protocol (headers and response
codes) in much the same way that Varnish does with its built-in gzip
support:
* Compression filters (the built-in ``"br"`` filter, or a custom
filter created from the ``encoder`` constructor) are not executed
for a backend response that already has a ``Content-Encoding``
header.
* Decompression filters (built-in ``"unbr"`` or a custom ``decoder``
filter) are not executed unless the backend response has the
``Content-Encoding`` header set to ``br``.
* When a compression filter is applied, the VFP sets the
``Content-Encoding`` response header to ``br``; this header value
may appear in a client response, and it may be cached with the
response. The value ``Accept-Encoding`` is also added to the
``Vary`` response header.
* When a decompression filter is applied, the ``Content-Encoding``
response header is removed.
* Any ``Content-Length`` response header fetched from the backend is
removed -- Varnish may set a new ``Content-Length`` header in a
client response, set to the changed size of the response body. When
streaming is enabled, Varnish sends the client response for a fetch
with chunked encoding (and hence with no ``Content-Length`` header).
* If the backend response has an ``ETag`` header, then the ``Etag``
value is "weakened" (prefixed with ``W/``), for weak validation
according to `RFC 7232`_ section 2.1. This is because the response
after de-/compression is not byte-identical with the fetched
response.
* The filters ignore partial responses to range requests (status 206
for "Partial Content").
Note that (unlike gzip) brotli compression does not work together with
Edge Side Includes (ESI). See `LIMITATIONS`_ below for a discussion of
this issue, and of possible solutions with VCL.
$Object encoder(PRIV_VCL, STRING name, BYTES bufffer=32768, INT quality=11,
BOOL large_win=0, INT lgwin=22,
ENUM {GENERIC, TEXT, FONT} mode=GENERIC)
XXX ...
Create a compression filter named ``name`` with custom parameters for
the brotli algorithm. The string given in ``name`` may then be used in
``beresp.filters``.
The default values for the parameters correspond to the defaults used
in the standard compression filter (so if you create a filter with all
defaults, it is functionally identical to the ``"br"`` filter).
The ``name`` MAY NOT be ``"br"`` or ``"unbr"``, or the same as any of
the standard filters built into Varnish: ``"esi"``, ``"esi_gzip"``,
``"gzip"``, ``"gunzip"`` or ``"testgunzip"``. The results are
undefined (and almost certainly not what you want) if you use a name
that is also used by another third-party VFP.
The parameters are:
* ``buffer`` (default 32KiB): Like the ``varnishd`` parameter
``gzip_buffer``, this is the size of the temporary internal buffer
used for compression. As with ``gzip_buffer``, a buffer size that is
too small results in more overhead, and if it is too large, then it
probably wastes memory.
* ``quality`` (default 11): sets a compression level, where 0
represents the fastest execution, and 11 represents the highest
compression rate (smallest compressed result). ``quality`` MUST be
>= 0 and <= 11.
* ``large_win`` (default ``false``): if ``true``, use Large Window
Brotli, which may yield better results for response bodies larger
than 16 MiB. The Large Window algorithm differs from the standard
algorithm; under certain circumstances, it may be necessary to use
the Large Window setting for both compression and decompression.
* ``lgwin`` (default 22): set the window size for the LZ77 algorithm.
The window size is (2 ^ ``lgwin`` - 16), and larger window sizes may
result in better compression rates. ``lgwin`` MUST be >= 10 and <=
24.
* ``mode`` (default ``GENERIC``): provide a hint about the expected
contents of the response body; the compression may benefit from
optimizations based on assumptions about the content. Possible
values are:
* ``GENERIC``: no assumptions are made about the content
* ``TEXT``: for UTF-8 formatted text
* ``FONT``: for the WOFF 2.0 font format (the original application
for brotli)
For more details about the parameters and their effects, see `RFC
7932`_ and the library documentation.
Example::
import brotli;
sub vcl_init {
# Create a compression filter with a 1MiB buffer, optimized
# for WOFF.
new br_woff = brotli.encoder("br_woff", buffer=1m, mode=WOFF);
}
sub vcl_backend_response {
# Use the custom filter for the WOFF media type.
if (beresp.http.Content-Type ~ "\bfont/woff2?\b") {
set beresp.filters = "br_woff";
}
}
$Method VOID .create_stats()
XXX ...
Create statistics, observable with a tool like ``varnishstat(1)``, for
the custom compression filter. These are the same as the counters
created for the standard ``"br"`` filter. See `STATISTICS`_ below for
details.
The ``.create_stats()`` method MAY NOT be called in any VCL subroutine
besides ``vcl_init``. If it is called in another subroutine, then VCL
failure is invoked -- ordinarily, the request fails with a "503 VCL
failed" response.
Example::
import brotli;
sub vcl_init {
# Create a custom compression filter with stats.
new mybr = brotli.encoder("mybr", buffer=64k);
mybr.create_stats();
}
$Object decoder(PRIV_VCL, STRING name, BYTES buffer=32768, BOOL large_win=0)
XXX ...
Create a decompression filter named ``name`` with custom parameters,
suitable for use in ``beresp.filters``.
As with the ``encoder`` object, the default values for the parameters
correspond to the defaults used in the standard ``"unbr"``
decompression filter.
Also as with ``"encoder``, the ``name`` MAY NOT be ``"br"`` or
``"unbr"``, or the same as any of Varnish's standard filters. It
SHOULD NOT be the same as a name used by another third-party VFP.
The parameters are:
* ``buffer`` (default 32KiB): like the ``buffer`` parameter for the
``encoder`` object, this is the size of the temporary internal
buffer used for decompression.
* ``large_win`` (default ``false``): if ``true``, use Large Window
Brotli. As noted above, it may be necessary to use the Large Window
setting for both compression and decompression.
Example::
import brotli;
sub vcl_init {
# Create a decompression filter with a small buffer size.
new unbr_small = brotli.decoder("unbr_small", buffer=1k);
}
sub vcl_backend_response {
# Use the custom filter when the URL begins with "/tiny"
if (bereq.url ~ "/tiny" && beresp.http.Content-Encoding ~ "br") {
set beresp.filters = "unbr_tiny";
}
}
$Method VOID .create_stats()
XXX ...
Create statistics for the custom decompression filter, like the
``.create_stats()`` method for ``encoder``. See `STATISTICS`_ below
for details.
As with ``encoder.create_stats()``, this method MAY NOT be called in
any VCL subroutine besides ``vcl_init``, otherwise VCL failure is invoked.
$Function STRING encoder_version()
Return the version string for the brotli encoder.
Return the version string for the brotli encoder library.
Example::
......@@ -90,7 +355,7 @@ Example::
$Function STRING decoder_version()
Return the version string for the brotli decoder.
Return the version string for the brotli decoder library.
Example::
......@@ -136,7 +401,9 @@ SEE ALSO
* vcl(7)
* varnishstat(1)
* VMOD source repository: https://code.uplex.de/uplex-varnish/libvfp-brotli
* brotli repository: https://github.com/google/brotli
* RFC 7932: https://tools.ietf.org/html/rfc7932
* brotli library repository: https://github.com/google/brotli
* RFC 7932 (brotli specification): https://tools.ietf.org/html/rfc7932
* RFC 7232 (HTTP conditional requests, with the specification for weak
validation): https://tools.ietf.org/html/rfc7232
$Event event
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment