README.rst 16 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14
..
.. NB:  This file is machine generated, DO NOT EDIT!
..
.. Edit vmod.vcc and run make instead
..

.. role:: ref(emphasis)

.. _vmod_blobdigest(3):

===============
vmod_blobdigest
===============

15 16 17
--------------------------------------------------
digests, checksums and hmacs for the VCL blob type
--------------------------------------------------
18 19 20 21 22 23 24 25 26

:Manual section: 3

SYNOPSIS
========

import blobdigest [from "path"] ;


27 28
::

29 30 31 32
  new OBJECT = blobdigest.digest(ENUM hash [, BLOB init])
  BOOL <obj>.update(BLOB)
  BLOB <obj>.final()

33 34 35 36 37 38 39 40
  BLOB blobdigest.hash(ENUM hash, BLOB msg)

  new OBJECT = blobdigest.hmac(ENUM hash, BLOB key)
  BLOB <obj>.hmac(BLOB msg)

DESCRIPTION
===========

Geoff Simmons's avatar
Geoff Simmons committed
41 42 43
This Varnish Module (VMOD) generates message digests, keyed-hash
message authentication codes (HMACs) and checksums using the VCL data
type BLOB, which may contain arbitrary data of any length.
44 45 46 47 48 49 50 51 52

Currently (in Varnish versions through 5.0), BLOBs may only be used in
VCL as arguments of VMOD functions, so this VMOD must be used in
combination with other VMODs. For example, the blobcode VMOD (see `SEE
ALSO`_) may be used to convert BLOBs using binary-to-text encodings,
to initialize data for this VMOD and to save its results. The
advantage of using BLOBs is that operations on the data are separated
from issues such as encoding.

53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128
digest object and hash function
-------------------------------

The VMOD provides messages digests by way of the ``digest`` object and
``hash`` function.

The interface for the ``digest`` object follows the
"init-update-final" idiom for cryptographic hashes. A digest context
is initialized in the constructor. The ``.update()`` method is used to
incrementally add data to be hashed -- calling ``.update(b1)`` and
then ``.update(b2)`` for BLOBs ``b1`` and ``b2`` has the same effect
as calling ``.update(b3)``, where ``b3`` is the concatenation of
``b1`` and ``b2``. The ``.final()`` method finalizes the message
digest and returns the result.

If an initial BLOB is provided in the ``digest`` constructor, and/or
if the ``.update()`` method is called one or more times in
``vcl_init``, then the resulting digest context is saved and re-used
for all subsequent method invocations on that object. This can be
used, for example, to efficiently compute digests for data that always
have a common prefix.

If the ``.final()`` method is invoked in ``vcl_init``, then the
resulting digest is computed and saved, and is retrieved for any
subsequent invocation of ``.final()``. This way, a hash can be
computed at initialization time and retrieved efficiently at runtime.

Otherwise, when ``.update()`` and ``.final()`` are invoked outside of
``vcl_init``, then their effects on the hash context and result have
"task" scope, meaning that they are valid within the current client or
backend context (which run in different threads in Varnish). For
example, when ``.update()`` is called in any of the ``vcl_backend_*``
subroutines, then it affects the digest result in any of the other
subroutines in the same backend transaction. This means that if
updates and finalizations are performed on the same object in both
client and backend transactions, then the results are independent of
another, and need not be the same.

The digest context of an object begins with the state as initialized
in ``vcl_init`` for each new client and backend transaction. So calls
to ``.update()`` and ``.final()`` outside of ``vcl_init`` do not
affect the state of the object in any other transaction.

Here are some examples::

  import blobdigest;
  import blobcode;
  import blob;

  sub vcl_init {
      # Create a BLOB consisting of the string "foo"
      new foo = blobcode.blob(IDENTITY, "foo");

      # Create a SHA256 context for messages with the prefix "foo"
      new fooprefix = blobdigest.digest(SHA256, foo.get());

      # This has the same effect
      new fooprefix2 = blobdigest.digest(SHA256);
      if (!fooprefix2.update(foo.get())) {
      	 return(fail);
      }

      # Compute and save the SHA256 hash for "foo"
      new foohash = blobdigest.digest(SHA256, foo.get());
      if (!blob.same(foohash.final(), foo.get())) {
      	 # This syntax is a workaround to allow .final() to be called,
	 # since VCL currently does not allow object methods to be
	 # called as procedures.
      }
  }

  sub vcl_recv {
      # Use the fooprefix object to get the hash of "foobar"
      if (!fooprefix.update(blobcode.decode(IDENTITY, "bar"))) {
      	 call do_error;
      }
129 130
      set req.http.Foobar-Hash = blobcode.encode(BASE64,
                                                 fooprefix.final());
131 132 133 134 135 136 137 138 139 140
  }

  sub vcl_backend_response {
      # Use the fooprefix object to get the hash of "foobaz".
      # The uses of the object in client and backend contexts have
      # no effect on each other, or on subsequent client or
      # backend transactions.
      if (!fooprefix.update(blobcode.decode(IDENTITY, "baz"))) {
      	 call do_error;
      }
141 142
      set req.http.Foobaz-Hash = blobcode.encode(BASE64,
                                                 fooprefix.final());
143 144 145 146
  }

  sub vcl_deliver {
      # Retrieve the SHA256 hash of "foo" computed in the constructor
147
      set req.http.Foo-Hash = blobcode.encode(BASE64, foohash.final());
148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164
  }

The ``hash()`` function computes the message digest for its argument
and returns the result. It is functionally equivalent to using a
``digest`` object::

  import blobdigest;
  import blobcode;

  sub vcl_init {
      # Create a SHA256 context
      new sha256 = blobdigest.digest(SHA256);
  }

  sub vcl_recv {
      # Get the SHA256 hash of "foo"
      set req.http.Foo-Hash-Functional
165 166 167
        = blobcode.encode(BASE64,
	                  blobdigest.hash(blobcode.decode(IDENTITY,
			                                  "foo")));
168 169 170 171 172 173

      # Same result using the object interface
      if (!sha256.update(blobcode.decode(IDENTITY, "foo"))) {
      	 call do_error;
      }
      set req.http.Foo-Hash-Object
Geoff Simmons's avatar
Geoff Simmons committed
174
        = blobcode.encode(BASE64, sha256.final());
175 176 177 178 179 180 181 182 183 184
  }

The ``hash()`` function makes for a somewhat less verbose interface,
and hence may be appropriate where no incremental updates are
necessary, and performance is less critical. But use of the ``digest``
object is more efficient, because the hash context is created once at
initialization time and then re-used.  The ``hash()`` function calls
initialization, update and finalization internally, so a new hash
context is created on every invocation.

185 186 187 188 189 190
HASH ALGORITHMS
===============

The ``hash`` enum in the following can have one of the following
values:

Geoff Simmons's avatar
Geoff Simmons committed
191
* ``CRC32`` (not for HMACs)
192 193 194 195 196 197 198 199 200 201 202
* ``MD5``
* ``SHA1``
* ``SHA224``
* ``SHA256``
* ``SHA384``
* ``SHA512``
* ``SHA3_224``
* ``SHA3_256``
* ``SHA3_384``
* ``SHA3_512``

203 204 205
CONTENTS
========

206 207 208
* digest(ENUM {CRC32,MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512}, BLOB)
* BLOB hash(ENUM {CRC32,MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512}, BLOB)
* hmac(ENUM {MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512}, BLOB)
209 210
* STRING version()

211 212
.. _obj_digest:

213 214
digest
------
215

216
::
217

218
	new OBJ = digest(ENUM {CRC32,MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512} hash, BLOB init=0)
219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242

Initialize a message digest context for the algorithm ``hash``, and
optionally update it with ``init``. If ``init`` is left out, then an
empty initial context is created.

If an ``init`` BLOB is provided, then the message digests computed
from this object result with ``init`` prepended before any BLOBs added
by the ``.update()`` method.

Example::

  import blobdigest;
  import blobcode;

  sub vcl_init {
      # Create an empty digest context for SHA3_256
      new sha3_256 = blobdigest.digest(SHA3_256);

      # Create a digest context for SHA512, and add "foo"
      # as a "prefix" for all other messages to be hashed.
      new foo = blobcode.blob(IDENTITY, "foo")
      new sha512 = blobdigest.digest(SHA512, foo.get());
  }

243 244
.. _func_digest.update:

245 246 247 248
digest.update
-------------

::
249 250 251

	BOOL digest.update(BLOB)

252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302
Incrementally add the BLOB to the digest context of this object.
Returns ``true`` if and only if the operation was successful.

As described above: if a digest object is updated in ``vcl_init``,
then the updated context is valid for all subsequent uses of the
object. Otherwise, the updated context is valid only for the current
task (client or backend transaction).

This method MAY NOT be called after ``.final()`` has been called
for the same object, either in ``vcl_init`` or in the current task.

The method fails and returns ``false`` if the BLOB is NULL, or if it
is called after ``.final()``. If it fails in ``vcl_init``, the VCL
load will fail with an error message. If it fails in any other VCL
subroutine, an error message is emitted to the Varnish log with the
``VCL_Error`` tag, and the message digest context is unchanged.

Example::

  import blobdigest;
  import blobcode;

  sub vcl_init {
      # Create a digest context for SHA512, and add "foo"
      # as a "prefix" for all other messages to be hashed.
      new f = blobcode.blob(IDENTITY, "f")
      new oo = blobcode.blob(IDENTITY, "oo")
      new sha512 = blobdigest.digest(SHA512);
      if (!sha512.update(f.get())) {
      	 return(fail);
      }
      if (!sha512.update(oo.get())) {
      	 return(fail);
      }

      # Create an empty digest context for SHA3_256
      new sha3_256 = blobdigest.digest(SHA3_256);
  }

  sub vcl_recv {
      # Update the SHA3_256 digest in the current client transaction
      if (!sha3_256.update(blobcode.blob(IDENTITY, "bar"))) {
      	 call do_client_error;
  }

  sub vcl_backend_fetch {
      # Update the SHA3_256 digest in the current backend transaction
      if (!sha3_256.update(blobcode.blob(IDENTITY, "baz"))) {
      	 call do_backend_error;
  }

303 304
.. _func_digest.final:

305 306 307 308
digest.final
------------

::
309 310 311

	BLOB digest.final()

312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367 368 369 370 371 372 373 374 375 376 377 378 379 380 381 382 383 384 385 386 387 388 389 390 391
Finalize the message digest and return the result.

If ``.final()`` is called in ``vcl_init``, then the message digest is
computed and saved, and returned for all subsequent calls. If it is
called in any other VCL subroutine, then the result is saved on the
first call, and returned for all other invocations of ``.final()`` in
the same task scope (that is, in the same client or backend
transaction).

As with ``.update()``, calling ``.final()`` outside of ``vcl_init``
only affects the state of the object in the current client or backend
transaction.

Example::

  import blobdigest;
  import blobcode;
  import blob;

  sub vcl_init {
      # Compute and save the SHA512 hash for "foo"
      new foo = blobcode.blob(IDENTITY, "foo")
      new foohash = blobdigest.digest(SHA512, foo.get());
      if (!blob.same(foohash.final(), foo.get())) {
      	 # As above, this is a workaround to call final(),
	 # which cannot be called as a procedure in VCL.
      }

      # Create an empty digest context for SHA3_256
      new sha3_256 = blobdigest.digest(SHA3_256);
  }

  sub vcl_recv {
      # Retrieve the hash for "foo" computed in vcl_init
      set req.http.Foo-Hash
        = blobcode.encode(BASE64, foohash.final());

      # Compute the base64-encoded SHA3_256 hash for "bar"
      if (!sha3_256.update(blobcode.decode(IDENTITY, "bar"))) {
      	 call do_client_error;
      }
      set req.http.Bar-Hash-Base64
        = blocbcode.decode(BASE64, sha3_256.final());
  }

  sub vcl_backend_fetch {
      # Compute the base64-encoded SHA3_256 hash for "baz"
      if (!sha3_256.update(blobcode.decode(IDENTITY, "baz"))) {
      	 call do_client_error;
      }
      set bereq.http.Baz-Hash-Base64
        = blocbcode.encode(BASE64, sha3_256.final());
  }

  sub vcl_backend_response {
      # Retrieve the message digest computed in vcl_backend_fetch
      # and get its hex encoding
      set beresp.http.Baz-Hash-Hex
        = blocbcode.encode(HEXLC, sha3_256.final());
      set beresp.http.Baz-Hash-Base64 = bereq.http.Baz-Hash-Base64
  }

  sub vcl_deliver {
      # Retrieve the message digest computed in vcl_recv and get
      # its hex encoding
      set resp.http.Bar-Hash-Hex
        = blocbcode.encode(HEXLC, sha3_256.final());
      set resp.http.Bar-Hash-Base64 = req.http.Bar-Hash-Base64;
      set resp.http.Foo-Hash = req.http.Foo-Hash;
  }

  # If there was a backend fetch (not a cache hit), then this VCL
  # creates the following response headers:
  #
  # Foo-Hash:        base64-encoded SHA512 hash of "foo"
  # Bar-Hash-Base64: base64-encoded SHA3_256 hash of "bar"
  # Bar-Hash-Hex:    hex-encoded SHA3_256 hash of "bar"
  # Baz-Hash-Base64: base64-encoded SHA3_256 hash of "baz"
  # Baz-Hash-Hex:    hex-encoded SHA3_256 hash of "bar"

392 393
.. _func_hash:

394 395 396 397
hash
----

::
398

399
	BLOB hash(ENUM {CRC32,MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512} hash, BLOB msg)
400 401 402 403 404 405 406 407 408 409 410 411 412 413 414

Returns the message digest for ``msg`` as specified by ``hash``.

Example::

	import blobdigest;
	import blobcode;

	# Decode the base64-encoded string, generate its SHA256 hash,
	# and save the hex-encoded result in a header.
	set req.http.SHA256
	    = blobcode.encode(HEXUC,
	                      blobdigest.hash(SHA256,
			                      blobcode.decode(BASE64, "Zm9v"));

415 416
.. _obj_hmac:

417 418
hmac
----
419

420
::
421

422
	new OBJ = hmac(ENUM {MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512} hash, BLOB key)
423

424 425 426 427
Creates an object that generates HMACs based on the digest algorithm
``hash`` and the given ``key``.

Example::
428

429 430 431 432 433 434 435 436 437
	import blobdigest;
	import blobcode;

	# Create a key from the base64-encoded string, and use
	# the result to initialize an hmac object.
	sub vcl_init {
		new key = blobcode.blob(BASE64, "a2V5");
		new hmac = blobdigest.hmac(SHA256, key.get());
	}
438 439 440

.. _func_hmac.hmac:

441 442 443 444
hmac.hmac
---------

::
445 446 447

	BLOB hmac.hmac(BLOB msg)

448 449
Returns the HMAC for ``msg`` based on the key and hash algorithm
provided in the constructor.
450

451
Example::
452

453 454
	# Import a VMOD that tests BLOBs for equality
	import blob;
455

456 457 458 459
	# Check if request header HMAC matches the HMAC for
	# the hex-encoded data in request header Msg.
	# Respond with 401 Not Authorized if the HMAC doesn't check.
	sub vcl_recv {
460
	    if (!blob.equal(blobcode.decode(HEX, req.http.HMAC),
461 462 463
	                    hmac.hmac(blobcode.decode(HEX, req.http.Msg)))) {
		return(synth(401));
	}
464

465 466
.. _func_version:

467 468 469 470
version
-------

::
471 472 473

	STRING version()

474 475 476 477 478 479 480 481 482
Returns the version string for this VMOD.

Example::

	std.log("Using VMOD blobdigest version " + blobdigest.version());

REQUIREMENTS
============

483 484 485
This version of the VMOD requires Varnish version 5.0.0. See the
source repository for versions that are compatible with other versions
of Varnish.
486

487 488 489
The gcc compiler and Perl 5 are required for the build. For the
self-tests invoked by ``make check``, the VMODs ``blobcode`` and
``blob`` must be installed (see `SEE ALSO`_).
490

491 492 493
LIMITATIONS
===========

494 495
For operations outside of ``vcl_init``, the VMOD allocates memory for
BLOBs and other internal structures in Varnish workspace.  If its
496 497 498 499 500
methods or functions fail, as indicated by "out of space" messages in
the Varnish log (with the ``VCL_Error`` tag), then you will need to
increase the varnishd parameters ``workspace_client`` and/or
``workspace_backend``.

501 502 503
For operations invoked in ``vcl_init``, the VMOD allocates heap
memory, and hence is only limited by available RAM.

504 505 506 507 508
INSTALLATION
============

See `INSTALL.rst <INSTALL.rst>`_ in the source repository.

509 510 511 512 513 514 515 516 517 518
AUTHOR
======

* Geoffrey Simmons <geoff@uplex.de>

UPLEX Nils Goroll Systemoptimierung

Cryptographic code is adapted from librhash by Aleksey Kravchenko,
which is in the public domain (see `LICENSE <LICENSE>`_).

Geoff Simmons's avatar
Geoff Simmons committed
519 520 521 522 523 524 525
CONTRIBUTING
============

See `CONTRIBUTING.rst <CONTRIBUTING.rst>`_ in the source repository
for notes on contributing source code and documentation, raising
issues, and for developer guidelines.

526 527
SEE ALSO
========
528

529 530
* varnishd(1)
* vcl(7)
531
* source repository: https://code.uplex.de/uplex-varnish/libvmod-blobdigest
532 533
* VMOD blobcode: https://code.uplex.de/uplex-varnish/libvmod-blobcode
* VMOD blob: https://code.uplex.de/uplex-varnish/libvmod-blob
534
* RHash:
535

536 537 538 539 540 541
  * https://github.com/rhash/RHash
  * http://rhash.anz.ru/

COPYRIGHT
=========

542 543 544 545 546 547 548 549 550
::

  Copyright (c) 2016 UPLEX Nils Goroll Systemoptimierung
  All rights reserved
 
  Author: Geoffrey Simmons <geoffrey.simmons@uplex.de>
 
  See LICENSE
 
551