README.rst 16.8 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14
..
.. NB:  This file is machine generated, DO NOT EDIT!
..
.. Edit vmod.vcc and run make instead
..

.. role:: ref(emphasis)

.. _vmod_blobdigest(3):

===============
vmod_blobdigest
===============

15 16 17
--------------------------------------------------
digests, checksums and hmacs for the VCL blob type
--------------------------------------------------
18 19 20 21 22 23 24 25 26

:Manual section: 3

SYNOPSIS
========

import blobdigest [from "path"] ;


27 28
::

29 30 31 32
  new OBJECT = blobdigest.digest(ENUM hash [, BLOB init])
  BOOL <obj>.update(BLOB)
  BLOB <obj>.final()

33 34 35 36 37
  BLOB blobdigest.hash(ENUM hash, BLOB msg)

  new OBJECT = blobdigest.hmac(ENUM hash, BLOB key)
  BLOB <obj>.hmac(BLOB msg)

Geoff Simmons's avatar
Geoff Simmons committed
38 39
  BLOB blobdigest.hmacf(ENUM hash, BLOB key, BLOB msg)

40 41 42
DESCRIPTION
===========

Geoff Simmons's avatar
Geoff Simmons committed
43 44 45
This Varnish Module (VMOD) generates message digests, keyed-hash
message authentication codes (HMACs) and checksums using the VCL data
type BLOB, which may contain arbitrary data of any length.
46

Nils Goroll's avatar
Nils Goroll committed
47 48 49 50
This vmod is intended for use together with the blob vmod from the
Varnish-Cache distribution for binary-to-text encodings, to initialize
data and to save results. The advantage of using BLOBs is that
operations on the data are separated from issues such as encoding.
51

52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97
digest object and hash function
-------------------------------

The VMOD provides messages digests by way of the ``digest`` object and
``hash`` function.

The interface for the ``digest`` object follows the
"init-update-final" idiom for cryptographic hashes. A digest context
is initialized in the constructor. The ``.update()`` method is used to
incrementally add data to be hashed -- calling ``.update(b1)`` and
then ``.update(b2)`` for BLOBs ``b1`` and ``b2`` has the same effect
as calling ``.update(b3)``, where ``b3`` is the concatenation of
``b1`` and ``b2``. The ``.final()`` method finalizes the message
digest and returns the result.

If an initial BLOB is provided in the ``digest`` constructor, and/or
if the ``.update()`` method is called one or more times in
``vcl_init``, then the resulting digest context is saved and re-used
for all subsequent method invocations on that object. This can be
used, for example, to efficiently compute digests for data that always
have a common prefix.

If the ``.final()`` method is invoked in ``vcl_init``, then the
resulting digest is computed and saved, and is retrieved for any
subsequent invocation of ``.final()``. This way, a hash can be
computed at initialization time and retrieved efficiently at runtime.

Otherwise, when ``.update()`` and ``.final()`` are invoked outside of
``vcl_init``, then their effects on the hash context and result have
"task" scope, meaning that they are valid within the current client or
backend context (which run in different threads in Varnish). For
example, when ``.update()`` is called in any of the ``vcl_backend_*``
subroutines, then it affects the digest result in any of the other
subroutines in the same backend transaction. This means that if
updates and finalizations are performed on the same object in both
client and backend transactions, then the results are independent of
another, and need not be the same.

The digest context of an object begins with the state as initialized
in ``vcl_init`` for each new client and backend transaction. So calls
to ``.update()`` and ``.final()`` outside of ``vcl_init`` do not
affect the state of the object in any other transaction.

Here are some examples::

  import blobdigest;
98
  import blob;
99 100 101

  sub vcl_init {
      # Create a BLOB consisting of the string "foo"
102
      new foo = blob.blob(IDENTITY, "foo");
103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123

      # Create a SHA256 context for messages with the prefix "foo"
      new fooprefix = blobdigest.digest(SHA256, foo.get());

      # This has the same effect
      new fooprefix2 = blobdigest.digest(SHA256);
      if (!fooprefix2.update(foo.get())) {
      	 return(fail);
      }

      # Compute and save the SHA256 hash for "foo"
      new foohash = blobdigest.digest(SHA256, foo.get());
      if (!blob.same(foohash.final(), foo.get())) {
      	 # This syntax is a workaround to allow .final() to be called,
	 # since VCL currently does not allow object methods to be
	 # called as procedures.
      }
  }

  sub vcl_recv {
      # Use the fooprefix object to get the hash of "foobar"
Nils Goroll's avatar
Nils Goroll committed
124
      if (!fooprefix.update(blob.decode(encoded="bar"))) {
125 126
      	 call do_error;
      }
Nils Goroll's avatar
Nils Goroll committed
127 128
      set req.http.Foobar-Hash = blob.encode(encoding=BASE64,
                                             blob=fooprefix.final());
129 130 131 132 133 134 135
  }

  sub vcl_backend_response {
      # Use the fooprefix object to get the hash of "foobaz".
      # The uses of the object in client and backend contexts have
      # no effect on each other, or on subsequent client or
      # backend transactions.
Nils Goroll's avatar
Nils Goroll committed
136
      if (!fooprefix.update(blob.decode(encoded="baz"))) {
137 138
      	 call do_error;
      }
Nils Goroll's avatar
Nils Goroll committed
139 140
      set req.http.Foobaz-Hash = blob.encode(encoding=BASE64,
                                             blob=fooprefix.final());
141 142 143 144
  }

  sub vcl_deliver {
      # Retrieve the SHA256 hash of "foo" computed in the constructor
Nils Goroll's avatar
Nils Goroll committed
145 146
      set req.http.Foo-Hash = blob.encode(encoding=BASE64,
                                          blob=foohash.final());
147 148 149 150 151 152 153
  }

The ``hash()`` function computes the message digest for its argument
and returns the result. It is functionally equivalent to using a
``digest`` object::

  import blobdigest;
154
  import blob;
155 156 157 158 159 160 161 162 163

  sub vcl_init {
      # Create a SHA256 context
      new sha256 = blobdigest.digest(SHA256);
  }

  sub vcl_recv {
      # Get the SHA256 hash of "foo"
      set req.http.Foo-Hash-Functional
Nils Goroll's avatar
Nils Goroll committed
164 165 166
        = blob.encode(encoding=BASE64,
	              blob=blobdigest.hash(blob.decode(encoded=
			                               "foo")));
167 168

      # Same result using the object interface
Nils Goroll's avatar
Nils Goroll committed
169
      if (!sha256.update(blob.decode(encoded="foo"))) {
170 171 172
      	 call do_error;
      }
      set req.http.Foo-Hash-Object
Nils Goroll's avatar
Nils Goroll committed
173
        = blob.encode(encoding=BASE64, blob=sha256.final());
174 175 176 177 178 179 180 181 182 183
  }

The ``hash()`` function makes for a somewhat less verbose interface,
and hence may be appropriate where no incremental updates are
necessary, and performance is less critical. But use of the ``digest``
object is more efficient, because the hash context is created once at
initialization time and then re-used.  The ``hash()`` function calls
initialization, update and finalization internally, so a new hash
context is created on every invocation.

184 185 186 187 188 189
HASH ALGORITHMS
===============

The ``hash`` enum in the following can have one of the following
values:

Geoff Simmons's avatar
Geoff Simmons committed
190
* ``CRC32`` (not for HMACs)
191 192 193 194 195 196 197 198 199 200 201
* ``MD5``
* ``SHA1``
* ``SHA224``
* ``SHA256``
* ``SHA384``
* ``SHA512``
* ``SHA3_224``
* ``SHA3_256``
* ``SHA3_384``
* ``SHA3_512``

202 203 204
CONTENTS
========

205 206 207
* digest(ENUM {CRC32,MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512}, BLOB)
* BLOB hash(ENUM {CRC32,MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512}, BLOB)
* hmac(ENUM {MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512}, BLOB)
Geoff Simmons's avatar
Geoff Simmons committed
208
* BLOB hmacf(ENUM {MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512}, BLOB, BLOB)
209 210
* STRING version()

211 212
.. _obj_digest:

213 214
digest
------
215

216
::
217

218
	new OBJ = digest(ENUM {CRC32,MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512} hash, BLOB init=0)
219 220 221 222 223 224 225 226 227 228 229 230

Initialize a message digest context for the algorithm ``hash``, and
optionally update it with ``init``. If ``init`` is left out, then an
empty initial context is created.

If an ``init`` BLOB is provided, then the message digests computed
from this object result with ``init`` prepended before any BLOBs added
by the ``.update()`` method.

Example::

  import blobdigest;
231
  import blob;
232 233 234 235 236 237 238

  sub vcl_init {
      # Create an empty digest context for SHA3_256
      new sha3_256 = blobdigest.digest(SHA3_256);

      # Create a digest context for SHA512, and add "foo"
      # as a "prefix" for all other messages to be hashed.
239
      new foo = blob.blob(IDENTITY, "foo")
240 241 242
      new sha512 = blobdigest.digest(SHA512, foo.get());
  }

243 244
.. _func_digest.update:

245 246 247 248
digest.update
-------------

::
249 250 251

	BOOL digest.update(BLOB)

252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271
Incrementally add the BLOB to the digest context of this object.
Returns ``true`` if and only if the operation was successful.

As described above: if a digest object is updated in ``vcl_init``,
then the updated context is valid for all subsequent uses of the
object. Otherwise, the updated context is valid only for the current
task (client or backend transaction).

This method MAY NOT be called after ``.final()`` has been called
for the same object, either in ``vcl_init`` or in the current task.

The method fails and returns ``false`` if the BLOB is NULL, or if it
is called after ``.final()``. If it fails in ``vcl_init``, the VCL
load will fail with an error message. If it fails in any other VCL
subroutine, an error message is emitted to the Varnish log with the
``VCL_Error`` tag, and the message digest context is unchanged.

Example::

  import blobdigest;
272
  import blob;
273 274 275 276

  sub vcl_init {
      # Create a digest context for SHA512, and add "foo"
      # as a "prefix" for all other messages to be hashed.
277 278
      new f = blob.blob(IDENTITY, "f")
      new oo = blob.blob(IDENTITY, "oo")
279 280 281 282 283 284 285 286 287 288 289 290 291 292
      new sha512 = blobdigest.digest(SHA512);
      if (!sha512.update(f.get())) {
      	 return(fail);
      }
      if (!sha512.update(oo.get())) {
      	 return(fail);
      }

      # Create an empty digest context for SHA3_256
      new sha3_256 = blobdigest.digest(SHA3_256);
  }

  sub vcl_recv {
      # Update the SHA3_256 digest in the current client transaction
293
      if (!sha3_256.update(blob.blob(IDENTITY, "bar"))) {
294 295 296 297 298
      	 call do_client_error;
  }

  sub vcl_backend_fetch {
      # Update the SHA3_256 digest in the current backend transaction
299
      if (!sha3_256.update(blob.blob(IDENTITY, "baz"))) {
300 301 302
      	 call do_backend_error;
  }

303 304
.. _func_digest.final:

305 306 307 308
digest.final
------------

::
309 310 311

	BLOB digest.final()

312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327
Finalize the message digest and return the result.

If ``.final()`` is called in ``vcl_init``, then the message digest is
computed and saved, and returned for all subsequent calls. If it is
called in any other VCL subroutine, then the result is saved on the
first call, and returned for all other invocations of ``.final()`` in
the same task scope (that is, in the same client or backend
transaction).

As with ``.update()``, calling ``.final()`` outside of ``vcl_init``
only affects the state of the object in the current client or backend
transaction.

Example::

  import blobdigest;
328
  import blob;
329 330 331

  sub vcl_init {
      # Compute and save the SHA512 hash for "foo"
332
      new foo = blob.blob(IDENTITY, "foo")
333 334 335 336 337 338 339 340 341 342 343 344 345
      new foohash = blobdigest.digest(SHA512, foo.get());
      if (!blob.same(foohash.final(), foo.get())) {
      	 # As above, this is a workaround to call final(),
	 # which cannot be called as a procedure in VCL.
      }

      # Create an empty digest context for SHA3_256
      new sha3_256 = blobdigest.digest(SHA3_256);
  }

  sub vcl_recv {
      # Retrieve the hash for "foo" computed in vcl_init
      set req.http.Foo-Hash
Nils Goroll's avatar
Nils Goroll committed
346
        = blob.encode(encoding=BASE64, blob=foohash.final());
347 348

      # Compute the base64-encoded SHA3_256 hash for "bar"
Nils Goroll's avatar
Nils Goroll committed
349
      if (!sha3_256.update(blob.decode(encoded="bar"))) {
350 351 352
      	 call do_client_error;
      }
      set req.http.Bar-Hash-Base64
Nils Goroll's avatar
Nils Goroll committed
353
        = blocbcode.decode(encoding=BASE64, blob=sha3_256.final());
354 355 356 357
  }

  sub vcl_backend_fetch {
      # Compute the base64-encoded SHA3_256 hash for "baz"
Nils Goroll's avatar
Nils Goroll committed
358
      if (!sha3_256.update(blob.decode(encoded="baz"))) {
359 360 361
      	 call do_client_error;
      }
      set bereq.http.Baz-Hash-Base64
Nils Goroll's avatar
Nils Goroll committed
362
        = blocbcode.encode(encoding=BASE64, blob=sha3_256.final());
363 364 365 366 367 368
  }

  sub vcl_backend_response {
      # Retrieve the message digest computed in vcl_backend_fetch
      # and get its hex encoding
      set beresp.http.Baz-Hash-Hex
369
        = blocbcode.encode(HEX, LOWER, sha3_256.final());
370 371 372 373 374 375 376
      set beresp.http.Baz-Hash-Base64 = bereq.http.Baz-Hash-Base64
  }

  sub vcl_deliver {
      # Retrieve the message digest computed in vcl_recv and get
      # its hex encoding
      set resp.http.Bar-Hash-Hex
377
        = blocbcode.encode(HEX, LOWER, sha3_256.final());
378 379 380 381 382 383 384 385 386 387 388 389 390
      set resp.http.Bar-Hash-Base64 = req.http.Bar-Hash-Base64;
      set resp.http.Foo-Hash = req.http.Foo-Hash;
  }

  # If there was a backend fetch (not a cache hit), then this VCL
  # creates the following response headers:
  #
  # Foo-Hash:        base64-encoded SHA512 hash of "foo"
  # Bar-Hash-Base64: base64-encoded SHA3_256 hash of "bar"
  # Bar-Hash-Hex:    hex-encoded SHA3_256 hash of "bar"
  # Baz-Hash-Base64: base64-encoded SHA3_256 hash of "baz"
  # Baz-Hash-Hex:    hex-encoded SHA3_256 hash of "bar"

391 392
.. _func_hash:

393 394 395 396
hash
----

::
397

398
	BLOB hash(ENUM {CRC32,MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512} hash, BLOB msg)
399 400 401 402 403 404

Returns the message digest for ``msg`` as specified by ``hash``.

Example::

	import blobdigest;
405
	import blob;
406 407 408 409

	# Decode the base64-encoded string, generate its SHA256 hash,
	# and save the hex-encoded result in a header.
	set req.http.SHA256
410
	    = blob.encode(HEX, UPPER,
411
	                      blobdigest.hash(SHA256,
Nils Goroll's avatar
Nils Goroll committed
412
			                      blob.decode(decoding=BASE64, encoded="Zm9v"));
413

414 415
.. _obj_hmac:

416 417
hmac
----
418

419
::
420

421
	new OBJ = hmac(ENUM {MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512} hash, BLOB key)
422

423 424 425 426
Creates an object that generates HMACs based on the digest algorithm
``hash`` and the given ``key``.

Example::
427

428
	import blobdigest;
429
	import blob;
430 431 432 433

	# Create a key from the base64-encoded string, and use
	# the result to initialize an hmac object.
	sub vcl_init {
434
		new key = blob.blob(BASE64, "a2V5");
435 436
		new hmac = blobdigest.hmac(SHA256, key.get());
	}
437 438 439

.. _func_hmac.hmac:

440 441 442 443
hmac.hmac
---------

::
444 445 446

	BLOB hmac.hmac(BLOB msg)

447 448
Returns the HMAC for ``msg`` based on the key and hash algorithm
provided in the constructor.
449

450
Example::
451

452 453
	# Import a VMOD that tests BLOBs for equality
	import blob;
454

455 456 457 458
	# Check if request header HMAC matches the HMAC for
	# the hex-encoded data in request header Msg.
	# Respond with 401 Not Authorized if the HMAC doesn't check.
	sub vcl_recv {
459 460
	    if (!blob.equal(blob.decode(HEX, req.http.HMAC),
	                    hmac.hmac(blob.decode(HEX, req.http.Msg)))) {
461 462
		return(synth(401));
	}
463

Geoff Simmons's avatar
Geoff Simmons committed
464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482 483 484 485 486
.. _func_hmacf:

hmacf
-----

::

	BLOB hmacf(ENUM {MD5,SHA1,SHA224,SHA256,SHA384,SHA512,SHA3_224,SHA3_256,SHA3_384,SHA3_512} hash, BLOB key, BLOB msg)

Returns the HMAC for ``msg`` as specified by ``hash`` and the ``key``.

As with the ``digest`` object and ``hash`` function, the use of the
``hmac`` object is likely to be more efficient than the ``hmacf``
function, because the internal cryptographic state of the HMAC for a
given key is pre-computed in the object constructor. So if the key is
fixed and known at initialization time, then you should use the
``hmac`` object. The ``hmacf`` function should only be used if the key
is not known at runtime, or, for example, should be changed without
requiring a VCL reload.

Example::

	import blobdigest;
487
	import blob;
Geoff Simmons's avatar
Geoff Simmons committed
488 489 490 491 492

	# Decode the base64-encoded string as a HMAC key, compute the
	# SHA512 HMAC of the Msg header, and save the hex-encoded
	# result in a header.
	set req.http.HMAC
493
	    = blob.encode(HEX, UPPER,
Geoff Simmons's avatar
Geoff Simmons committed
494
	                      blobdigest.hmacf(SHA512,
Nils Goroll's avatar
Nils Goroll committed
495 496
			                       blob.decode(decoding=BASE64, encoded="Zm9v"),
					       blob.decode(encoded=
Geoff Simmons's avatar
Geoff Simmons committed
497 498
					                       req.http.Msg)));

499 500
.. _func_version:

501 502 503 504
version
-------

::
505 506 507

	STRING version()

508 509 510 511 512 513 514 515 516
Returns the version string for this VMOD.

Example::

	std.log("Using VMOD blobdigest version " + blobdigest.version());

REQUIREMENTS
============

Nils Goroll's avatar
Nils Goroll committed
517
This version of the VMOD requires Varnish since version 5.2. See the
518 519
source repository for versions that are compatible with other versions
of Varnish.
520

Nils Goroll's avatar
Nils Goroll committed
521
The gcc compiler and Perl 5 are required for the build.
522

523 524 525
LIMITATIONS
===========

526 527
For operations outside of ``vcl_init``, the VMOD allocates memory for
BLOBs and other internal structures in Varnish workspace.  If its
528 529 530 531 532
methods or functions fail, as indicated by "out of space" messages in
the Varnish log (with the ``VCL_Error`` tag), then you will need to
increase the varnishd parameters ``workspace_client`` and/or
``workspace_backend``.

533 534 535
For operations invoked in ``vcl_init``, the VMOD allocates heap
memory, and hence is only limited by available RAM.

536 537 538 539 540
INSTALLATION
============

See `INSTALL.rst <INSTALL.rst>`_ in the source repository.

541 542 543 544 545 546 547 548 549 550
AUTHOR
======

* Geoffrey Simmons <geoff@uplex.de>

UPLEX Nils Goroll Systemoptimierung

Cryptographic code is adapted from librhash by Aleksey Kravchenko,
which is in the public domain (see `LICENSE <LICENSE>`_).

Geoff Simmons's avatar
Geoff Simmons committed
551 552 553 554 555 556 557
CONTRIBUTING
============

See `CONTRIBUTING.rst <CONTRIBUTING.rst>`_ in the source repository
for notes on contributing source code and documentation, raising
issues, and for developer guidelines.

558 559
SEE ALSO
========
560

561 562
* varnishd(1)
* vcl(7)
563
* source repository: https://code.uplex.de/uplex-varnish/libvmod-blobdigest
564
* RHash:
565

566 567 568 569 570 571
  * https://github.com/rhash/RHash
  * http://rhash.anz.ru/

COPYRIGHT
=========

572 573 574 575 576 577 578 579 580
::

  Copyright (c) 2016 UPLEX Nils Goroll Systemoptimierung
  All rights reserved
 
  Author: Geoffrey Simmons <geoffrey.simmons@uplex.de>
 
  See LICENSE
 
581