#-
# Copyright 2020 UPLEX Nils Goroll Systemoptimierung
# All rights reserved.
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
#    notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions and the following disclaimer in the
#    documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.

$Module tus 3 "Varnish tus Module"
$Event event
$Prefix tus

DESCRIPTION
===========

.. _tus: https://tus.io/protocols/resumable-upload.html

This vmod implements a `tus`_ proxy which collects uploads on the
varnish server to send them to a backend in one go for storage. It
does not implement any permanent storage itself.

Besides the basic resumable uploads specified as the `tus`_ core
protocol, all currently defined extensions are supported, in
particular concatenation uploads.

This vmod implements methods to be called from VCL and requires a
common call pattern for normal operation. When used this way, this
vmod handles all client interaction via synthetic responses until an
upload is complete. For the client request concluding the upload, it
modifies the request into a single ``PUT`` and provides a single
request body to be sent to a backend.

By this VCL integration, http behaviour remains highly customizable.

Persistent Storage
------------------

Uploads using this module are persistent across Varnish and system
restarts, given that the ``basedir`` argument to `tus.server()`_
specifies a directory which remains intact across Varnish and system
restarts, respectively.

When the first `tus.server()`_ is initialized, all files in
``basedir`` which match the vmod_tus naming convention (prefixed by
``tus_``) are checked and reused if they appear consistent and have
not yet expired.

Concat final uploads are not restored, but parts are.

CORS
----

The `xserver.synth()`_ and `xserver.deliver()`_ methods will set the
right CORS headers allowing all Origins, that is,
``Access-Control-Allow-Origin`` set to ``Origin``. As any other
header, the CORS headers can be changed after the call to
`xserver.synth()`_, and should be to restrict Origin access.

Hashes
------

.. _vmod_blobdigest: https://code.uplex.de/uplex-varnish/libvmod-blobdigest

This vmod supports checking and generating content hashes if
`vmod_blob` (bundled with varnish-cache) and `vmod_blobdigest`_ are

* installed at build time
* installed in exactly the same version at run time
* and imported to the VCL before ``tus``, like so::

	import blob;
	import blobdigest;
	import tus;

Given these preconditions are true, thus vmod supports the following
hashes as ``Tus-Checksum-Algorithm`` and as the ``name_hash`` argument
to `tus.server()`_:

	* crc32
	* icrc32
	* md5
	* rs
	* sha1
	* sha224
	* sha256
	* sha384
	* sha3_224
	* sha3_256
	* sha3_384
	* sha3_512
	* sha512


$Object server(STRING schemeauth, BYTES max=1073741824,
    DURATION expires=86400, [STRING basedir],
    [STRING name_hash])

Declare a tus server at the given ``schemeauth``, which is the
protocol and servername, like ``https://tus.io``.

Tus servers are shared across VCLs. Each server has its own namespace.

The ``max`` argument (defaulting to 1GB) specifies the maximum upload
size supported by the server. The actual maximum upload size supported
as reportde in the ``Tus-Max-Size`` header will be that value capped
by the available disk space in the ``basedir``. Notice that as the
available space might vary inbetween requests, so will the reported
space, and uploads might still fail. Thus the ``max`` argument is
recommended to be chosen sufficiently small compared to the available
disk space and disk space be monitored.

The ``expires`` argument (defaulting to 1 day) specifies the duration
until unfinished uploads expire and are removed. The expiry gets
extended by that duration with every operation on a file.

.. XXX warm event? per VCL object?

The ``schemeauth``, ``max`` and ``expires`` values for an already defined
server *can* be changed. The value used in the vcl *loaded* last
applies.

The optional ``basedir`` argument can be used to define a directory
under which this server's objects are stored. This directory or its
parent directory should exist.

Sharing basedirs between servers is not supported. An attempting to
change the ``basedir`` for a server already defined through a
different VCL is an error.

The optional ``name_hash`` argument can be used to instruct the tus
server to use the hex encoding of a hash over the content the upload
as the last part of the url when constructing the backend ``PUT``. For
``Upload-Concat: final`` uploads, this name is also used as the
``Location`` returned to the client.

$Method BOOL .recv([STRING url], [STRING id])

Process a tus request.

If the `url` argument is not given, it is taken from ``req.url``.

The optional `id` argument allows to override the id assigned by the
tus server for a create (HTTP ``POST``). Notice that, while id is
normally guaranteed to be unique for the currently active uploads, a
``POST`` may fail for a duplicate `id` argument. `id` has no relevance
for any request but ``POST``.

If the return value is ``true``, the request body has been replaced
with an object ready to be sent to the backend, so a backend call
should be made to send data to storage.

If the reuturn value is ``false``, a synthetic reponse has to be
generated, so ``return(synth(code))`` should be returned such that
`xserver.synth()`_ will be called for ``code``.

Thus, the common call pattern is::

	sub vcl_recv {
		if (mytus.recv()) {
			return (pass);
		} else {
			return (synth(4200));
		}
	}

Must be called from ``vcl_recv {}``

$Method BOOL .deliver()

Generate a response to a tus request for which content was
stored. Must be called from ``vcl_deliver {}`` after `xserver.recv()`_
was called from ``vcl_recv {}``.

$Method BOOL .synth()

Generate a synthetic response to a tus request. Must be called from
``vcl_synth {}`` after `xserver.recv()`_ was called from ``vcl_recv {}``.

For the example above, the code would be::

	sub vcl_synth {
		if (resp.status == 4200) {
			mytus.synth();
			return (deliver);
		}
	}

$Method BOOL .done([STRING location])

Mark the upload as done. For simple (non concat) uploads, this frees
all storage except for information on the upload URL itself.

If the optional *location* string is provided, a ``Content-Location:``
response header with that location will be added to all future
responses querying the object.

For any ``GET`` access, a 301 response with the *location* in the
``Location:`` response header is generated.

May only be called from client methods. For anything but final concat
or single (non-concat) uploads, this operation is a noop.

$Method BOOL .has_metadata(STRING key)

Return true if *key* is present in the metadata.

Only available if vmod_blob is available (see `Hashes` _) and on the
client side after `xserver.recv()`_ was called from ``vcl_recv {}``.

$Method BLOB .metadata(STRING key)

Extract *key* from metadata and return the corresponding value decoded.

Only available if vmod_blob is available (see `Hashes` _) and on the
client side after `xserver.recv()`_ was called from ``vcl_recv {}``.

$Method VOID .sync()

For lowest latencies, vmod_tus stores data in files opened without any
additional options affecting caching such that the operating system is
free to write data delayed. This could, in turn, lead to upload data
and/or state to be lost or reverted to an earlier state when the
system crashes.

When the `xserver.sync()`_ method is called, an `fdatasync(2)` call is
issued on the file backing the current upload, such that its current
state is flushed to stable storage (given that the underlying file
does provide stable storage).


SEE ALSO
========vcl\(7),varnishd\(1)