#- # Copyright 2020 UPLEX Nils Goroll Systemoptimierung # All rights reserved. # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions # are met: # 1. Redistributions of source code must retain the above copyright # notice, this list of conditions and the following disclaimer. # 2. Redistributions in binary form must reproduce the above copyright # notice, this list of conditions and the following disclaimer in the # documentation and/or other materials provided with the distribution. # # THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND # ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE # ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE # FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL # DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS # OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) # HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT # LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY # OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF # SUCH DAMAGE. $Module tus 3 "Varnish tus Module" $Event event $Prefix tus DESCRIPTION =========== .. _tus: https://tus.io/protocols/resumable-upload.html This vmod implements a `tus`_ proxy which collects uploads on the varnish server to send them to a backend in one go for storage. It does not implement any permanent storage itself. Besides the basic resumable uploads specified as the `tus`_ core protocol, all currently defined extensions are supported, in particular concatenation uploads. This vmod implements methods to be called from VCL and requires a common call pattern for normal operation. When used this way, this vmod handles all client interaction via synthetic responses until an upload is complete. For the client request concluding the upload, it modifies the request into a single ``PUT`` and provides a single request body to be sent to a backend. By this VCL integration, http behaviour remains highly customizable. Persistent Storage ------------------ Uploads using this module are persistent across Varnish and system restarts, given that the ``basedir`` argument to `tus.server()`_ specifies a directory which remains intact across Varnish and system restarts, respectively. When the first `tus.server()`_ is initialized, all files in ``basedir`` which match the vmod_tus naming convention (prefixed by ``tus_``) are checked and reused if they appear consistent and have not yet expired. Concat final uploads are not restored, but parts are. CORS ---- The `xserver.synth()`_ and `xserver.deliver()`_ methods will set the right CORS headers allowing all Origins, that is, ``Access-Control-Allow-Origin`` set to ``Origin``. As any other header, the CORS headers can be changed after the call to `xserver.synth()`_, and should be to restrict Origin access. Hashes ------ .. _vmod_blobdigest: https://code.uplex.de/uplex-varnish/libvmod-blobdigest This vmod supports checking and generating content hashes if `vmod_blob` (bundled with varnish-cache) and `vmod_blobdigest`_ are * installed at build time * installed in exactly the same version at run time * and imported to the VCL before ``tus``, like so:: import blob; import blobdigest; import tus; Given these preconditions are true, thus vmod supports the following hashes as ``Tus-Checksum-Algorithm`` and as the ``name_hash`` argument to `tus.server()`_: * crc32 * icrc32 * md5 * rs * sha1 * sha224 * sha256 * sha384 * sha3_224 * sha3_256 * sha3_384 * sha3_512 * sha512 $Object server(STRING schemeauth, BYTES max=1073741824, DURATION expires=86400, [STRING basedir], [STRING name_hash]) Declare a tus server at the given ``schemeauth``, which is the protocol and servername, like ``https://tus.io``. Tus servers are shared across VCLs. Each server has its own namespace. The ``max`` argument (defaulting to 1GB) specifies the maximum upload size supported by the server. The actual maximum upload size supported as reportde in the ``Tus-Max-Size`` header will be that value capped by the available disk space in the ``basedir``. Notice that as the available space might vary inbetween requests, so will the reported space, and uploads might still fail. Thus the ``max`` argument is recommended to be chosen sufficiently small compared to the available disk space and disk space be monitored. The ``expires`` argument (defaulting to 1 day) specifies the duration until unfinished uploads expire and are removed. The expiry gets extended by that duration with every operation on a file. .. XXX warm event? per VCL object? The ``schemeauth``, ``max`` and ``expires`` values for an already defined server *can* be changed. The value used in the vcl *loaded* last applies. The optional ``basedir`` argument can be used to define a directory under which this server's objects are stored. This directory or its parent directory should exist. Sharing basedirs between servers is not supported. An attempting to change the ``basedir`` for a server already defined through a different VCL is an error. The optional ``name_hash`` argument can be used to instruct the tus server to use the hex encoding of a hash over the content the upload as the last part of the url when constructing the backend ``PUT``. For ``Upload-Concat: final`` uploads, this name is also used as the ``Location`` returned to the client. $Method BOOL .recv([STRING url], [STRING id]) Process a tus request. If the `url` argument is not given, it is taken from ``req.url``. The optional `id` argument allows to override the id assigned by the tus server for a create (HTTP ``POST``). Notice that, while id is normally guaranteed to be unique for the currently active uploads, a ``POST`` may fail for a duplicate `id` argument. `id` has no relevance for any request but ``POST``. If the return value is ``true``, the request body has been replaced with an object ready to be sent to the backend, so a backend call should be made to send data to storage. If the reuturn value is ``false``, a synthetic reponse has to be generated, so ``return(synth(code))`` should be returned such that `xserver.synth()`_ will be called for ``code``. Thus, the common call pattern is:: sub vcl_recv { if (mytus.recv()) { return (pass); } else { return (synth(4200)); } } Must be called from ``vcl_recv {}`` $Method BOOL .deliver() Generate a response to a tus request for which content was stored. Must be called from ``vcl_deliver {}`` after `xserver.recv()`_ was called from ``vcl_recv {}``. $Method BOOL .synth() Generate a synthetic response to a tus request. Must be called from ``vcl_synth {}`` after `xserver.recv()`_ was called from ``vcl_recv {}``. For the example above, the code would be:: sub vcl_synth { if (resp.status == 4200) { mytus.synth(); return (deliver); } } $Method BOOL .done([STRING location]) Mark the upload as done. For simple (non concat) uploads, this frees all storage except for information on the upload URL itself. If the optional *location* string is provided, a ``Content-Location:`` response header with that location will be added to all future responses querying the object. For any ``GET`` access, a 301 response with the *location* in the ``Location:`` response header is generated. May only be called from client methods. For anything but final concat or single (non-concat) uploads, this operation is a noop. $Method BOOL .has_metadata(STRING key) Return true if *key* is present in the metadata. Only available if vmod_blob is available (see `Hashes` _) and on the client side after `xserver.recv()`_ was called from ``vcl_recv {}``. $Method BLOB .metadata(STRING key) Extract *key* from metadata and return the corresponding value decoded. Only available if vmod_blob is available (see `Hashes` _) and on the client side after `xserver.recv()`_ was called from ``vcl_recv {}``. $Method VOID .sync() For lowest latencies, vmod_tus stores data in files opened without any additional options affecting caching such that the operating system is free to write data delayed. This could, in turn, lead to upload data and/or state to be lost or reverted to an earlier state when the system crashes. When the `xserver.sync()`_ method is called, an `fdatasync(2)` call is issued on the file backing the current upload, such that its current state is flushed to stable storage (given that the underlying file does provide stable storage). SEE ALSO ========vcl\(7),varnishd\(1)