..
.. NB:  This file is machine generated, DO NOT EDIT!
..
.. Edit ./vmod_file.vcc and run make instead
..

.. role:: ref(emphasis)

=========
vmod_file
=========

-----------------------------------------------------------------
Varnish Module for reading files that may be updated at intervals
-----------------------------------------------------------------

:Manual section: 3


SYNOPSIS
========

::

  import file;

  # File reader object
  new <obj> = file.reader(STRING name [, STRING path] [, DURATION ttl])

  # File contents
  STRING <obj>.get()
  VOID <obj>.synth()
  BLOB <obj>.blob()
  VOID <obj>.check()

  # File metadata
  BYTES <obj>.size()
  TIME <obj>.mtime()
  DURATION <obj>.next_check()
  BOOL <obj>.deleted()

  # Digests that change when the file changes
  BLOB <obj>.id()
  BLOB <obj>.sha256()

  # Error status
  BOOL <obj>.error()
  STRING <obj>.errmsg()

  # VMOD version
  STRING file.version()

DESCRIPTION
===========

.. _Varnish: http://www.varnish-cache.org/

.. _VCL: http://varnish-cache.org/docs/trunk/reference/vcl.html

VMOD file is a `Varnish`_ module for reading the contents of a file
and caching its contents, returning the contents for use in the
Varnish Configuration Language (`VCL`_), and checking if the file has
changed after specified time intervals elapse.

.. _VMOD std: https://varnish-cache.org/docs/trunk/reference/vmod_std.html

.. _std.fileread(): https://varnish-cache.org/docs/trunk/reference/vmod_std.html#std-fileread

.. |std.fileread()| replace:: ``std.fileread()``

`VMOD std`_, provided with the Varnish distribution, has the function
|std.fileread()|_, which reads the contents of a file exactly once on
the first invocation, caches the contents, and returns the cached
contents on every subsequent invocation. The cache is static, so the
cached file contents do not change when VCL is reloaded, or at any
other time until Varnish stops.  This minimizes file I/O during
request/response transactions, which is incurred only on the first
invocation. But it means that changed file contents do not become
available in VCL with |std.fileread()|_ unless Varnish is re-started.

VMOD file provides a `reader object`_, which caches file contents
during the invocation of ``vcl_init`` (hence at VCL load time). The
object is provided by default with a time interval, for which the
concept "TTL" (time to live) is re-applied. When a TTL is set, the
file is periodically checked for changes in a background thread, every
time the TTL elapses. If the file has changed, then the file cache is
reloaded with its new contents, which then become available for
subsequent accesses via the reader object in VCL::

  import file;

  sub vcl_init {
	# Cache the contents of a file, and check it for changes
	# every 30 seconds.
	new rdr = file.reader("/path/to/myfile", ttl=30s);
  }

  sub vcl_recv {
	# Read the cached contents of the file. If the file was found
	# to have changed, then the new contents are returned by .get().
	set req.http.Myfile = rdr.get();
  }

The content cache takes the form of a memory-mapping of the file, see
``mmap(2)``. This has some consequences that are discussed in the
sections `File deletion and file updates`_ and `LIMITATIONS`_ below.

Since the update checks run in the background, the file I/O that the
checks require is not incurred during any request/response
transaction.

.. _vcl.state: https://varnish-cache.org/docs/trunk/reference/varnish-cli.html#vcl-state-configname-auto-cold-warm

.. _varnish-cli(7): https://varnish-cache.org/docs/trunk/reference/varnish-cli.html

.. |vcl.state| replace:: ``vcl.state``

.. _vcl.use: https://varnish-cache.org/docs/trunk/reference/varnish-cli.html#vcl-use-configname-label

.. |vcl.use| replace:: ``vcl.use``

When a VCL instance transitions to the cold state, the file cache is
unmapped, and file update checks for any of the instance's objects are
suspended (see |vcl.state|_ in `varnish-cli(7)`_). When it transitions
back to the warm state (which also happens during an invocation of
|vcl.use|_ if the VCL had previously been cold), then the files are
immediately re-read and cached, and the update checks in the
background resume at the TTL interval.

File deletion and file updates
------------------------------

POSIX mandates that mmap(2) adds a reference for the file, which is
not removed until the file is unmapped. In particular, it is not
removed when the file is deleted -- the mapping continues to access
the file's contents, even after deletion. (In that case, the file is
not physically removed, but is no longer accessible by name in the
filesystem.)

.. |.deleted()| replace:: ``.deleted()``

For this reason, if an update check finds that the file has been
deleted, *it is not considered an error*, provided that the file has
already been mapped. (It is an error if the file does not exist at
initialization.) The file is considered unchanged, and the cached
contents remain valid, at least until the next check. The
|.deleted()|_ method of the `reader object`_ can be used in VCL to
detect this situation.

POSIX leaves unspecified whether changes in the underlying file
immediately become visible in the memory mapping. On a system like
Linux, changes are immediately visible, and hence will be reflected
immediately be the VMOD.  While this may seem ideal for getting fast
updates in VCL, it is in fact problematic:

* File writes are not atomic; so the VMOD may return partial and
  inconsistent contents for the file.

* If the changed file is longer than the originally mapped file, the
  portion that is longer than the original file is not
  mapped. Contents returned by the VMOD will appear truncated.

For these reasons, this is a reliable method to update a file:

* Delete the file

* Write the new contents to a new file of the same name (same path
  location)

This is the *only* method for updating files that the VMOD supports.

After the deletion step, the previously cached contents remain valid.
When the next update check detects the change performed by the second
step, the new contents are mapped, and become available in their
correct form via the VMOD.

Other means of updating the file might "happen" to work, some of the
time. But if not, it is not considered a bug of the VMOD. The VMOD
works as designed *only* if the two-step procedure for updating files
is followed.

.. _reader object:

.. _file.reader():

new xreader = file.reader(STRING name, STRING path, DURATION ttl, BOOL log_checks, BOOL enable_sha256)
------------------------------------------------------------------------------------------------------

::

   new xreader = file.reader(
      STRING name,
      STRING path="/usr/local/etc/varnish:/usr/local/share/varnish/vcl:/etc/varnish:/usr/share/varnish/vcl",
      DURATION ttl=120,
      BOOL log_checks=0,
      BOOL enable_sha256=0
   )

Create an object to read and cache the contents of the file named
``name``, and optionally check the file for changes at the interval
``ttl``. ``name`` MAY NOT be the empty string. If ``ttl`` is set to
0s, then no periodic checks are performed. ``ttl`` MAY NOT be < 0s.
By default, ``ttl`` is 120 seconds.

If ``name`` denotes an absolute path (beginning with ``/``), then the
file at that path is read. Otherwise, the file is searched for in the
directories given in the colon-separated string ``path``. The file
MUST fulfill the following conditions:

* The file MUST be accessible to the owner of the Varnish child
  process.

* The process owner MUST have read permissions on the file.

* The file MUST be a regular file, or a symbolic link pointing to a
  regular file.

If any of these are not true of ``name``, or if no such file is found
on the ``path``, then the VCL load fails with an error message.

.. _vcl_path: https://varnish-cache.org/docs/trunk/reference/varnishd.html#vcl-path

.. |vcl_path| replace:: ``vcl_path``

The default value of ``path`` combines the default values of the
varnishd parameter |vcl_path|_ for development builds (installed in
``/usr/local``) and production deployments (installed in ``/usr``),
with the development directories first. ``path`` MAY NOT be the empty
string.

If there is an error finding or reading the file, then the VCL load
fails with a message describing the error. If the read succeeds, then
the file contents are cached, and are available via the reader
object's methods.

If initialization succeeds and ``ttl`` > 0s, then update checks begin
at that interval. A file is considered to have changed if any of its
``stat(2)`` fields ``mtime``, ``dev`` or ``ino`` change. As discussed
above, the file is considered unchanged if the update check finds the
the file has been deleted, provided that it has already been mapped;
then the previously cached contents continue to be valid. If the file
has changed when a check is performed, it is re-read and the new
contents are cached, for access via the object's methods.

.. |.sha256()| replace:: ``.sha256()``

.. |.id()| replace:: ``.id()``

If ``enable_sha256`` is ``true``, then a SHA256 digest is computed for
the file's contents at initialization time, and whenever the file is
determined to have changed. This is done by the update check, not
during any request/response transaction, and hence does not affect
response times. The digest may then be retrieved with the |.sha256()|_
method. If ``enable_sha256`` is ``false``, then no SHA256 hash is
computed, and it is an error to invoke |.sha256()|_.  Since the digest
may be expensive to compute, especially for large files, it may be
advantageous to leave SHA256 disabled, and use the less costly
|.id()|_ method instead; see the documentation of |.sha256()|_ for a
discussion.  By default, ``enable_sha256`` is ``false``.

If an error is encountered when a check attempts to re-read the file,
then subsequent method calls attempting to access the contents invoke
VCL failure (see `ERRORS`_ below), with the ``VCL_Error`` message in
the Varnish log describing the error.

Checks continue at the ``ttl`` interval, regardless of any error. If
the next update check after an error succeeds (because the problem has
been fixed in the meantime), then the new contents are cached, and
object methods can access the contents successfully.

.. _vsl(7): https://varnish-cache.org/docs/trunk/reference/vsl.html

.. _vsl_mask: https://varnish-cache.org/docs/trunk/reference/varnishd.html#vsl-mask

.. |vsl_mask| replace:: ``vsl_mask``

.. _raw grouping: https://varnish-cache.org/docs/trunk/reference/vsl-query.html#grouping

.. _varnishlog(1): https://varnish-cache.org/docs/trunk/reference/varnishlog.html

If ``log_checks`` is ``true`` (default ``false``), then the activity
of update checks is logged in the Varnish log using the tag ``Debug``
(see `vsl(7)`_). By default, ``Debug`` logs are filtered from the
Varnish log; to see them, add ``Debug`` to the varnishd parameter
|vsl_mask|_, for example by invoking varnishd with
``-p vsl_mask=+Debug``. Since update checks do not happen during any
request/response transaction, they are logged with pseudo-XID 0, and
are only visible when the log is read with `raw grouping`_, for
example by invoking `varnishlog(1)`_ with ``-g raw``.

Regardless of the value of ``log_checks``, errors encountered during
update checks are logged with the tag ``Error``, also with XID 0 (and
hence visible in raw grouping). A message is always written to the log
with the ``Debug`` tag (using XID 0) if an update check finds that the
file has been deleted, but is already mapped (and hence is considered
unchanged).

Examples::

  sub vcl_init {
	# A reader for the file at the absolute path, using default
	# ttl=120s.
	new foo = file.reader("/path/to/foo");

	# A reader for the file on the default search path, with
	# update checks every five minutes.
	new synth_body = file.reader("synth_body.html", ttl=300s);

	# A reader for the file on the given search path, with
	# default TTL, logging for update checks, and SHA256 enabled.
	new bar = file.reader("bar", path="/var/run/d1:/var/run/d2",
	                       log_checks=true, enable_sha256=true);

	# A reader for the file with no update checks.
	new baz = file.reader("baz", ttl=0s);
  }

.. _xreader.get():

STRING xreader.get()
--------------------

Return the contents of the file specified in the constructor, as
currently cached. If the most recent update check encountered an
error, then VCL failure is invoked (see `ERRORS`_).

Example::

  sub vcl_deliver {
	set resp.http.Foo = foo.get();
  }

Take care if you use ``.get()`` to set a header, as in the example,
that the file contents do *not* end in a newline. If so, then the
newline appears after the header contents, resulting in an empty line
after the header. Since an empty line separates the headers from the
body in an HTTP message, this is very likely to result in an invalid
message.

.. _xreader.synth():

VOID xreader.synth()
--------------------

Generate a synthetic response body from the file contents. This method
may only be called in ``vcl_synth`` or ``vcl_backend_error``. Invokes
VCL failure if the most recent update check encountered an error, or
if invoked in any other VCL subroutine besides the two that are
permitted.

Example::

  sub vcl_synth {
	synth_body.synth();
  }

  sub vcl_backend_error {
	synth_body.synth();
  }

.. _xreader.blob():

BLOB xreader.blob()
-------------------

Return the file's contents as a BLOB. Invokes VCL failure if the most
recent update check encountered an error.

Example::

  import blob;

  # Set the backend response body to the hex-encoded contents of
  # the file. Also works for resp.body in vcl_synth.
  sub vcl_backend_error {
	set beresp.body = blob.encode(HEX, blob=synth_body.blob());
  }

.. _reader.error():

.. _xreader.check():

VOID xreader.check()
--------------------

Run a synchronous check of the file, and update the cached contents if
the file has been changed. This is the same check that runs
periodically in the background after the reader object is created.

Since the file check does not scale well, this method should be used
judiciously as a part of request processing. A reasonable use case is
an administrative request with restricted access, to be called as a
part of an automated process when the underlying file is changed, so
that the changes are reflected promptly.

Example::

  if (req.url == "/update-files") {
	# Assume that this ACL defines internal admin networks.
	# Return error status 403 Forbidden if the client IP doesn't
	# match.
	if (client.ip !~ admin_network) {
		return (synth(403));
	}
	# Internal admins may run a synchronous file check.
	rdr.check();
  }

.. _xreader.error():

BOOL xreader.error()
--------------------

Return true if and only if an error condition was determined the last
time the file was checked. This is a way to avoid VCL failure in error
conditions.

Example::

  if (rdr.error()) {
	call do_file_error_handling;
  }

.. _xreader.errmsg():

STRING xreader.errmsg()
-----------------------

Return the error message for any error condition determined the last
time the file was checked, or a message indicating that there was no
error.

Example::

  import std;

  if (rdr.error()) {
	std.log("rdr error: " + rdr.errmsg());
	call do_file_error_handling;
  }

.. _.deleted():

.. _xreader.deleted():

BOOL xreader.deleted()
----------------------

Return true if and only if the file was found to have been deleted the
last time the file was checked.

As discussed in `File deletion and file updates`_ above, this is not
an error condition, if the file had been previously mapped. Then the
previously cached contents continue to be valid.

Example::

  import std;

  if (rdr.deleted()) {
	std.log("file deleted, continuing with the current cached contents");
  }

.. _xreader.size():

BYTES xreader.size()
--------------------

Return the size of the file as currently cached. Invokes VCL failure
if the most recent update check encountered an error.

Example::

  # Use the cached synth body if non-empty, otherwise use the standard
  # Varnish Guru Meditation.
  if (synth_body.size() > 0B) {
	synth_body.synth();
  }

.. _xreader.mtime():

TIME xreader.mtime()
--------------------

Return the modification time of the file determined when it was mostly
recently checked. Invokes VCL failure if the most recent update check
encountered an error.

Example::

  # A VCL TIME is converted to a string as an HTTP date, so .mtime()
  # is suitable for the Last-Modified header. If the If-Modified-Since
  # request header is present and designates an earlier time than
  # .mtime(), send a 304 Not Modified response.

  import std;

  sub vcl_recv {
     # std.time() parses If-Modified-Since as a TIME. If the parse
     # fails, fall back to now (which is almost certainly not earlier
     # than the mtime).
     if (std.time(req.http.If-Modified-Since, now) < rdr.mtime()) {
	return (synth(304));
     }
     else {
	return (synth(200));
     }
  }

  sub vcl_synth {
      set resp.http.Last-Modified = rdr.mtime();
      if (resp.status == 200) {
	rdr.synth();
      }
      return(deliver);
  }

.. _xreader.next_check():

DURATION xreader.next_check()
-----------------------------

Return the time remaining until the next check will be performed.

Example::

  import std;

  # Set the downstream caching TTL to the time remaining until the
  # next update check.
  set resp.http.Cache-Control = "public, max-age="
	+ std.integer(duration=rdr.next_check());

.. _.id():

.. _xreader.id():

BLOB xreader.id()
-----------------

Returns a unique, opaque identifier for the state of the file as
determined when it was most recently checked. The ID changes if and
only if the file was found to have changed. Invokes VCL failure if the
most recent update check encountered an error.

Example::

  # Use the base64 encoding of .id() for the ETag response header, and
  # send a 304 response if the If-None-Match request header matches
  # it.

  import blob;

  sub vcl_recv {
     if (req.http.If-None-Match == blob.encode(BASE64, blob=rdr.id())) {
	return (synth(304));
     }
     else {
	return (synth(200));
     }
  }

  sub vcl_synth {
      set resp.http.ETag = blob.encode(BASE64, blob=rdr.id());
      if (resp.status == 200) {
	rdr.synth();
      }
      return(deliver);
  }

The contents of the BLOB returned by ``.id()`` are intentionally not
documented, and should not be relied on to extract information about
the file.

.. _.sha256():

.. _xreader.sha256():

BLOB xreader.sha256()
---------------------

Returns the SHA256 digest of the file's contents, if the
``enable_sha256`` flag was set to ``true`` in the constructor. Invokes
VCL failure if SHA256 was not enabled, or if the most recent update
check encountered an error.

Example::

  # Use .sha256() for the ETag and If-None-Match headers, as shown
  # above for .id().

  import blob;

  sub vcl_init {
     new rdr = file.reader("/path/to/file", enable_sha256=true);
  }

  sub vcl_recv {
     if (req.http.If-None-Match == blob.encode(BASE64, blob=rdr.sha256())) {
	return (synth(304));
     }
     else {
	return (synth(200));
     }
  }

  sub vcl_synth {
      set resp.http.ETag = blob.encode(BASE64, blob=rdr.sha256());
      if (resp.status == 200) {
	rdr.synth();
      }
      return(deliver);
  }

The ``.sha256()`` and ``.id()`` methods both return a digest for the
file, each of which changes when the file changes, and hence is
suitable for a header such as ``ETag``. In the typical case, the
``.id()`` method will lead to more efficient use of system resources.
This is because ``.id()`` is derived solely from the file's metadata
-- the fields obtained from ``stat(2)`` -- and when these change, the
file's contents typically have been changed as well.

If only the file metadata have been changed, but not the contents,
then the return value of ``.id()`` changes nevertheless. This may
happen for example if the modification time has been updated by
``touch(1)``, or if the file contents happen to be no different after
an update. In that case, using the ``.id()`` value for
``If-None-Match`` and ``ETag`` may lead to sending the response body
unnecessarily, when a 304 Not Modified response would have been
possible. The return value of ``.sha256()`` remains unchanged in such
a situation.

But if a change in the metadata without a change of the contents is
uncommon, then it makes more sense to disable SHA256 and use
``.id()``. Then the VMOD saves the effort of computing the SHA256
digest, and ``.id()`` is sufficient for a purpose such as ``ETag``.

.. _file.version():

STRING version()
----------------

Return the version string for this VMOD.

Example::

  std.log("Using VMOD file version: " + file.version());

ERRORS
======

Methods that access a file's cached contents invoke VCL failure if
there was an error during the most recent update check, just as if
``return(fail)`` had been invoked in VCL. This means that:

* If the error occurs during ``vcl_init`` (on the initial read of the
  file), then the VCL load fails with an error message.

* If the error occurs during any other subroutine besides
  ``vcl_synth``, then a ``VCL_Error`` message describing the problem
  is written to the log, and control is immediately directed to
  ``vcl_synth``. In ``vcl_synth``, the response status
  (``resp.status``) is set to 503, and the reason string
  (``resp.reason``) is set to ``"VCL failed"``.

* If the error happens during ``vcl_synth``, then the ``VCL_Error``
  message is written, ``vcl_synth`` is aborted. The response line
  ``"503 VCL failed"`` is set, but the client may just see connection
  reset.

.. |reader.error()| replace:: ``reader.error()``

The |reader.error()|_ may be used to detect errors, for example to
implement different error handling in VCL.

Errors that may be encountered on the initial read or update checks
include:

* The file cannot be opened for read. This is what will happen for
  typical file errors: the Varnish process cannot access the file, or
  the process owner does not have read permissions.

* The file does not exist at initialization time. As discussed above,
  this is not an error for an update check, if the file has already
  been mapped.

* The file is neither a regular file nor a symbolic link that points
  to a regular file.

* Any of the internal calls to map the file fail.

REQUIREMENTS
============

The VMOD currently requires the Varnish master branch, and is
compatible with Varnish version 6.3.0.

INSTALLATION
============

See `INSTALL.rst <INSTALL.rst>`_ in the source repository.

LIMITATIONS
===========

Cached file contents (mapped with mmap(2)) consume virtual memory
space.  This can become a burden if large files are cached, and/or if
they are cached by many VMOD objects in many VCL instances.

.. _vcl.discard: https://varnish-cache.org/docs/trunk/reference/varnish-cli.html#vcl-discard-configname-label

.. |vcl.discard| replace:: ``vcl.discard``

.. _vcl_cooldown: https://varnish-cache.org/docs/trunk/reference/varnishd.html#vcl-cooldown

.. |vcl_cooldown| replace:: ``vcl_cooldown``

File caches are unmapped and timers are suspended when the VCL
instance transitions to the cold state. Timers are deleted when the
VMOD's reader objects are finalized, which happens when the
|vcl.discard|_ command is used to unload VCL instances. When a new VCL
instance is made active (with ``vcl.use``), the previously active
instance enters the cold state after the period defined by the
varnishd parameter |vcl_cooldown|_ elapses (10 minutes in recent
Varnish versions), unless ``vcl.discard`` is invoked first. The cold
transition always takes place on VCL discard, if the instance is not
already cold.

So if you need to release the resources held by objects of this VMOD
in inactive VCL instances, consider shortening the ``vcl_cooldown``
period and/or automating the removal of old instances with
``vcl.discard``.  (Housekeeping with ``vcl.discard`` is a good
practice for Varnish that is often neglected.)

If the file unmappings and timer suspensions and deletions fail during
the cold transition or object finalization, error messages are written
to the Varnish log using the tag ``Error`` (visible with raw
grouping). While these errors are unlikely, if they do happen, they
may be indications of resource leaks. Consider monitoring the log for
such errors.

.. _VSL query: https://varnish-cache.org/docs/trunk/reference/vsl-query.html

Log messages from the VMOD begin with the prefix ``vmod file``. A `VSL
query`_ can be used to craft a `varnishlog(1)`_ invocation that
filters out the VMOD's messages::

  varnishlog -g raw -q 'Debug ~ "^vmod file" or Error ~ "^vmod file"'

It is platform-dependent whether file I/O is incurred during the first
request/response transactions that read file contents, or whether at
least some of the I/O work is done at initialization, and after file
contents are newly mapped by an update check. The VMOD provides a hint
that the mapped file contents may be used imminently (using
``posix_madvise(3)`` with ``WILLNEED``); the kernel may respond by
reading ahead in the file mapping. But that decision is left to the
kernel.

If SHA256 is enabled, then the digest is computed after the file is
mapped.  In that case, I/O for file reads is guaranteed to take place
before any request/response transaction.

SEE ALSO
========

* source repository website: https://code.uplex.de/uplex-varnish/libvmod-file
  * Mirror: https://gitlab.com/uplex/varnish/libvmod-file
* Varnish: http://www.varnish-cache.org/
* varnishd(1): http://varnish-cache.org/docs/trunk/reference/varnishd.html
* vcl(7): http://varnish-cache.org/docs/trunk/reference/vcl.html
* varnishlog(1): https://varnish-cache.org/docs/trunk/reference/varnishlog.html
* vsl(7): https://varnish-cache.org/docs/trunk/reference/vsl.html
* vsl-query(7): https://varnish-cache.org/docs/trunk/reference/vsl-query.html
* varnish-cli(7): https://varnish-cache.org/docs/trunk/reference/varnish-cli.html
* VMOD std: https://varnish-cache.org/docs/trunk/reference/vmod_std.html
* mmap(2)
* stat(2)

COPYRIGHT
=========

::

  Copyright (c) 2019 UPLEX Nils Goroll Systemoptimierung
  All rights reserved
 
  Author: Geoffrey Simmons <geoffrey.simmons@uplex.de>
 
  See LICENSE