README.rst 17.4 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14
..
.. NB:  This file is machine generated, DO NOT EDIT!
..
.. Edit vmod.vcc and run make instead
..

.. role:: ref(emphasis)

.. _vmod_blobdigest(3):

===============
vmod_blobdigest
===============

15 16 17
--------------------------------------------------
digests, checksums and hmacs for the VCL blob type
--------------------------------------------------
18 19 20 21 22 23

:Manual section: 3

SYNOPSIS
========

24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

::

   import blobdigest [from "path"] ;
   
   new xdigest = digest(ENUM, BLOB, ENUM)
  
      BOOL xdigest.update(BLOB)
  
      BLOB xdigest.final()
  
   BLOB hash(ENUM, BLOB)
  
   new xhmac = hmac(ENUM, BLOB)
  
      BLOB xhmac.hmac(BLOB)
  
   BLOB hmacf(ENUM, BLOB, BLOB)
  
   STRING version()
  
45 46


47 48
::

49
  new OBJECT = blobdigest.digest(ENUM hash [, BLOB init], ["TASK"|"TOP"])
50 51 52
  BOOL <obj>.update(BLOB)
  BLOB <obj>.final()

53 54 55 56 57
  BLOB blobdigest.hash(ENUM hash, BLOB msg)

  new OBJECT = blobdigest.hmac(ENUM hash, BLOB key)
  BLOB <obj>.hmac(BLOB msg)

Geoff Simmons's avatar
Geoff Simmons committed
58 59
  BLOB blobdigest.hmacf(ENUM hash, BLOB key, BLOB msg)

60 61 62
DESCRIPTION
===========

Geoff Simmons's avatar
Geoff Simmons committed
63 64 65
This Varnish Module (VMOD) generates message digests, keyed-hash
message authentication codes (HMACs) and checksums using the VCL data
type BLOB, which may contain arbitrary data of any length.
66

Nils Goroll's avatar
Nils Goroll committed
67 68 69 70
This vmod is intended for use together with the blob vmod from the
Varnish-Cache distribution for binary-to-text encodings, to initialize
data and to save results. The advantage of using BLOBs is that
operations on the data are separated from issues such as encoding.
71

72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117
digest object and hash function
-------------------------------

The VMOD provides messages digests by way of the ``digest`` object and
``hash`` function.

The interface for the ``digest`` object follows the
"init-update-final" idiom for cryptographic hashes. A digest context
is initialized in the constructor. The ``.update()`` method is used to
incrementally add data to be hashed -- calling ``.update(b1)`` and
then ``.update(b2)`` for BLOBs ``b1`` and ``b2`` has the same effect
as calling ``.update(b3)``, where ``b3`` is the concatenation of
``b1`` and ``b2``. The ``.final()`` method finalizes the message
digest and returns the result.

If an initial BLOB is provided in the ``digest`` constructor, and/or
if the ``.update()`` method is called one or more times in
``vcl_init``, then the resulting digest context is saved and re-used
for all subsequent method invocations on that object. This can be
used, for example, to efficiently compute digests for data that always
have a common prefix.

If the ``.final()`` method is invoked in ``vcl_init``, then the
resulting digest is computed and saved, and is retrieved for any
subsequent invocation of ``.final()``. This way, a hash can be
computed at initialization time and retrieved efficiently at runtime.

Otherwise, when ``.update()`` and ``.final()`` are invoked outside of
``vcl_init``, then their effects on the hash context and result have
"task" scope, meaning that they are valid within the current client or
backend context (which run in different threads in Varnish). For
example, when ``.update()`` is called in any of the ``vcl_backend_*``
subroutines, then it affects the digest result in any of the other
subroutines in the same backend transaction. This means that if
updates and finalizations are performed on the same object in both
client and backend transactions, then the results are independent of
another, and need not be the same.

The digest context of an object begins with the state as initialized
in ``vcl_init`` for each new client and backend transaction. So calls
to ``.update()`` and ``.final()`` outside of ``vcl_init`` do not
affect the state of the object in any other transaction.

Here are some examples::

  import blobdigest;
118
  import blob;
119 120 121

  sub vcl_init {
      # Create a BLOB consisting of the string "foo"
122
      new foo = blob.blob(IDENTITY, "foo");
123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143

      # Create a SHA256 context for messages with the prefix "foo"
      new fooprefix = blobdigest.digest(SHA256, foo.get());

      # This has the same effect
      new fooprefix2 = blobdigest.digest(SHA256);
      if (!fooprefix2.update(foo.get())) {
      	 return(fail);
      }

      # Compute and save the SHA256 hash for "foo"
      new foohash = blobdigest.digest(SHA256, foo.get());
      if (!blob.same(foohash.final(), foo.get())) {
      	 # This syntax is a workaround to allow .final() to be called,
	 # since VCL currently does not allow object methods to be
	 # called as procedures.
      }
  }

  sub vcl_recv {
      # Use the fooprefix object to get the hash of "foobar"
Nils Goroll's avatar
Nils Goroll committed
144
      if (!fooprefix.update(blob.decode(encoded="bar"))) {
145 146
      	 call do_error;
      }
Nils Goroll's avatar
Nils Goroll committed
147 148
      set req.http.Foobar-Hash = blob.encode(encoding=BASE64,
                                             blob=fooprefix.final());
149 150 151 152 153 154 155
  }

  sub vcl_backend_response {
      # Use the fooprefix object to get the hash of "foobaz".
      # The uses of the object in client and backend contexts have
      # no effect on each other, or on subsequent client or
      # backend transactions.
Nils Goroll's avatar
Nils Goroll committed
156
      if (!fooprefix.update(blob.decode(encoded="baz"))) {
157 158
      	 call do_error;
      }
Nils Goroll's avatar
Nils Goroll committed
159 160
      set req.http.Foobaz-Hash = blob.encode(encoding=BASE64,
                                             blob=fooprefix.final());
161 162 163 164
  }

  sub vcl_deliver {
      # Retrieve the SHA256 hash of "foo" computed in the constructor
Nils Goroll's avatar
Nils Goroll committed
165 166
      set req.http.Foo-Hash = blob.encode(encoding=BASE64,
                                          blob=foohash.final());
167 168 169 170 171 172 173
  }

The ``hash()`` function computes the message digest for its argument
and returns the result. It is functionally equivalent to using a
``digest`` object::

  import blobdigest;
174
  import blob;
175 176 177 178 179 180 181 182 183

  sub vcl_init {
      # Create a SHA256 context
      new sha256 = blobdigest.digest(SHA256);
  }

  sub vcl_recv {
      # Get the SHA256 hash of "foo"
      set req.http.Foo-Hash-Functional
Nils Goroll's avatar
Nils Goroll committed
184 185 186
        = blob.encode(encoding=BASE64,
	              blob=blobdigest.hash(blob.decode(encoded=
			                               "foo")));
187 188

      # Same result using the object interface
Nils Goroll's avatar
Nils Goroll committed
189
      if (!sha256.update(blob.decode(encoded="foo"))) {
190 191 192
      	 call do_error;
      }
      set req.http.Foo-Hash-Object
Nils Goroll's avatar
Nils Goroll committed
193
        = blob.encode(encoding=BASE64, blob=sha256.final());
194 195 196 197 198 199 200 201 202 203
  }

The ``hash()`` function makes for a somewhat less verbose interface,
and hence may be appropriate where no incremental updates are
necessary, and performance is less critical. But use of the ``digest``
object is more efficient, because the hash context is created once at
initialization time and then re-used.  The ``hash()`` function calls
initialization, update and finalization internally, so a new hash
context is created on every invocation.

204 205 206 207 208 209
HASH ALGORITHMS
===============

The ``hash`` enum in the following can have one of the following
values:

Geoff Simmons's avatar
Geoff Simmons committed
210
* ``CRC32`` (not for HMACs)
Nils Goroll's avatar
Nils Goroll committed
211 212
* ``ICRC32`` (not for HMACs): CRC32 variant with inverted initialization
  and result
213
* ``MD5``
214
* ``RS`` (not for HMACs)
215 216 217 218 219 220 221 222 223 224
* ``SHA1``
* ``SHA224``
* ``SHA256``
* ``SHA384``
* ``SHA512``
* ``SHA3_224``
* ``SHA3_256``
* ``SHA3_384``
* ``SHA3_512``

225

226 227
.. _obj_digest:

228 229
new xdigest = digest(ENUM, BLOB, ENUM)
--------------------------------------
230

231
::
232

233 234 235 236 237
   new xdigest = digest(
      ENUM {CRC32, ICRC32, MD5, RS, SHA1, SHA224, SHA256, SHA384, SHA512, SHA3_224, SHA3_256, SHA3_384, SHA3_512} hash,
      BLOB init=0,
      ENUM {TASK, TOP} scope=TASK
   )
238 239 240 241 242 243 244 245 246

Initialize a message digest context for the algorithm ``hash``, and
optionally update it with ``init``. If ``init`` is left out, then an
empty initial context is created.

If an ``init`` BLOB is provided, then the message digests computed
from this object result with ``init`` prepended before any BLOBs added
by the ``.update()`` method.

247 248 249 250 251 252 253
By default, the scope of the message digest context is ``TASK``:
method calls affect separate context states for the client and backend
side.

``TOP`` scope provides a context across all ESI subrequests, but is
available only on the client side.

254 255 256
Example::

  import blobdigest;
257
  import blob;
258 259 260 261 262 263 264

  sub vcl_init {
      # Create an empty digest context for SHA3_256
      new sha3_256 = blobdigest.digest(SHA3_256);

      # Create a digest context for SHA512, and add "foo"
      # as a "prefix" for all other messages to be hashed.
265
      new foo = blob.blob(IDENTITY, "foo")
266 267 268
      new sha512 = blobdigest.digest(SHA512, foo.get());
  }

269 270
.. _func_digest.update:

271 272
BOOL xdigest.update(BLOB)
-------------------------
273

274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293
Incrementally add the BLOB to the digest context of this object.
Returns ``true`` if and only if the operation was successful.

As described above: if a digest object is updated in ``vcl_init``,
then the updated context is valid for all subsequent uses of the
object. Otherwise, the updated context is valid only for the current
task (client or backend transaction).

This method MAY NOT be called after ``.final()`` has been called
for the same object, either in ``vcl_init`` or in the current task.

The method fails and returns ``false`` if the BLOB is NULL, or if it
is called after ``.final()``. If it fails in ``vcl_init``, the VCL
load will fail with an error message. If it fails in any other VCL
subroutine, an error message is emitted to the Varnish log with the
``VCL_Error`` tag, and the message digest context is unchanged.

Example::

  import blobdigest;
294
  import blob;
295 296 297 298

  sub vcl_init {
      # Create a digest context for SHA512, and add "foo"
      # as a "prefix" for all other messages to be hashed.
299 300
      new f = blob.blob(IDENTITY, "f")
      new oo = blob.blob(IDENTITY, "oo")
301 302 303 304 305 306 307 308 309 310 311 312 313 314
      new sha512 = blobdigest.digest(SHA512);
      if (!sha512.update(f.get())) {
      	 return(fail);
      }
      if (!sha512.update(oo.get())) {
      	 return(fail);
      }

      # Create an empty digest context for SHA3_256
      new sha3_256 = blobdigest.digest(SHA3_256);
  }

  sub vcl_recv {
      # Update the SHA3_256 digest in the current client transaction
315
      if (!sha3_256.update(blob.blob(IDENTITY, "bar"))) {
316 317 318 319 320
      	 call do_client_error;
  }

  sub vcl_backend_fetch {
      # Update the SHA3_256 digest in the current backend transaction
321
      if (!sha3_256.update(blob.blob(IDENTITY, "baz"))) {
322 323 324
      	 call do_backend_error;
  }

325

326
.. _func_digest.final:
327

328 329
BLOB xdigest.final()
--------------------
330

331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346
Finalize the message digest and return the result.

If ``.final()`` is called in ``vcl_init``, then the message digest is
computed and saved, and returned for all subsequent calls. If it is
called in any other VCL subroutine, then the result is saved on the
first call, and returned for all other invocations of ``.final()`` in
the same task scope (that is, in the same client or backend
transaction).

As with ``.update()``, calling ``.final()`` outside of ``vcl_init``
only affects the state of the object in the current client or backend
transaction.

Example::

  import blobdigest;
347
  import blob;
348 349 350

  sub vcl_init {
      # Compute and save the SHA512 hash for "foo"
351
      new foo = blob.blob(IDENTITY, "foo")
352 353 354 355 356 357 358 359 360 361 362 363 364
      new foohash = blobdigest.digest(SHA512, foo.get());
      if (!blob.same(foohash.final(), foo.get())) {
      	 # As above, this is a workaround to call final(),
	 # which cannot be called as a procedure in VCL.
      }

      # Create an empty digest context for SHA3_256
      new sha3_256 = blobdigest.digest(SHA3_256);
  }

  sub vcl_recv {
      # Retrieve the hash for "foo" computed in vcl_init
      set req.http.Foo-Hash
Nils Goroll's avatar
Nils Goroll committed
365
        = blob.encode(encoding=BASE64, blob=foohash.final());
366 367

      # Compute the base64-encoded SHA3_256 hash for "bar"
Nils Goroll's avatar
Nils Goroll committed
368
      if (!sha3_256.update(blob.decode(encoded="bar"))) {
369 370 371
      	 call do_client_error;
      }
      set req.http.Bar-Hash-Base64
Nils Goroll's avatar
Nils Goroll committed
372
        = blocbcode.decode(encoding=BASE64, blob=sha3_256.final());
373 374 375 376
  }

  sub vcl_backend_fetch {
      # Compute the base64-encoded SHA3_256 hash for "baz"
Nils Goroll's avatar
Nils Goroll committed
377
      if (!sha3_256.update(blob.decode(encoded="baz"))) {
378 379 380
      	 call do_client_error;
      }
      set bereq.http.Baz-Hash-Base64
Nils Goroll's avatar
Nils Goroll committed
381
        = blocbcode.encode(encoding=BASE64, blob=sha3_256.final());
382 383 384 385 386 387
  }

  sub vcl_backend_response {
      # Retrieve the message digest computed in vcl_backend_fetch
      # and get its hex encoding
      set beresp.http.Baz-Hash-Hex
388
        = blocbcode.encode(HEX, LOWER, sha3_256.final());
389 390 391 392 393 394 395
      set beresp.http.Baz-Hash-Base64 = bereq.http.Baz-Hash-Base64
  }

  sub vcl_deliver {
      # Retrieve the message digest computed in vcl_recv and get
      # its hex encoding
      set resp.http.Bar-Hash-Hex
396
        = blocbcode.encode(HEX, LOWER, sha3_256.final());
397 398 399 400 401 402 403 404 405 406 407 408 409
      set resp.http.Bar-Hash-Base64 = req.http.Bar-Hash-Base64;
      set resp.http.Foo-Hash = req.http.Foo-Hash;
  }

  # If there was a backend fetch (not a cache hit), then this VCL
  # creates the following response headers:
  #
  # Foo-Hash:        base64-encoded SHA512 hash of "foo"
  # Bar-Hash-Base64: base64-encoded SHA3_256 hash of "bar"
  # Bar-Hash-Hex:    hex-encoded SHA3_256 hash of "bar"
  # Baz-Hash-Base64: base64-encoded SHA3_256 hash of "baz"
  # Baz-Hash-Hex:    hex-encoded SHA3_256 hash of "bar"

410 411 412 413




414 415
.. _func_hash:

416 417
BLOB hash(ENUM, BLOB)
---------------------
418 419

::
420

421 422 423 424
   BLOB hash(
      ENUM {CRC32, ICRC32, MD5, RS, SHA1, SHA224, SHA256, SHA384, SHA512, SHA3_224, SHA3_256, SHA3_384, SHA3_512} hash,
      BLOB msg
   )
425 426 427 428 429 430

Returns the message digest for ``msg`` as specified by ``hash``.

Example::

	import blobdigest;
431
	import blob;
432 433 434 435

	# Decode the base64-encoded string, generate its SHA256 hash,
	# and save the hex-encoded result in a header.
	set req.http.SHA256
436
	    = blob.encode(HEX, UPPER,
437
	                      blobdigest.hash(SHA256,
Nils Goroll's avatar
Nils Goroll committed
438
			                      blob.decode(decoding=BASE64, encoded="Zm9v"));
439

440

441 442
.. _obj_hmac:

443 444
new xhmac = hmac(ENUM, BLOB)
----------------------------
445

446
::
447

448 449 450 451
   new xhmac = hmac(
      ENUM {MD5, SHA1, SHA224, SHA256, SHA384, SHA512, SHA3_224, SHA3_256, SHA3_384, SHA3_512} hash,
      BLOB key
   )
452

453 454 455 456
Creates an object that generates HMACs based on the digest algorithm
``hash`` and the given ``key``.

Example::
457

458
	import blobdigest;
459
	import blob;
460 461 462 463

	# Create a key from the base64-encoded string, and use
	# the result to initialize an hmac object.
	sub vcl_init {
464
		new key = blob.blob(BASE64, "a2V5");
465 466
		new hmac = blobdigest.hmac(SHA256, key.get());
	}
467 468 469

.. _func_hmac.hmac:

470 471
BLOB xhmac.hmac(BLOB msg)
-------------------------
472

473 474
Returns the HMAC for ``msg`` based on the key and hash algorithm
provided in the constructor.
475

476
Example::
477

478 479
	# Import a VMOD that tests BLOBs for equality
	import blob;
480

481 482 483 484
	# Check if request header HMAC matches the HMAC for
	# the hex-encoded data in request header Msg.
	# Respond with 401 Not Authorized if the HMAC doesn't check.
	sub vcl_recv {
485 486
	    if (!blob.equal(blob.decode(HEX, req.http.HMAC),
	                    hmac.hmac(blob.decode(HEX, req.http.Msg)))) {
487 488
		return(synth(401));
	}
489

490 491 492 493




Geoff Simmons's avatar
Geoff Simmons committed
494 495
.. _func_hmacf:

496 497
BLOB hmacf(ENUM, BLOB, BLOB)
----------------------------
Geoff Simmons's avatar
Geoff Simmons committed
498 499 500

::

501 502 503 504 505
   BLOB hmacf(
      ENUM {MD5, SHA1, SHA224, SHA256, SHA384, SHA512, SHA3_224, SHA3_256, SHA3_384, SHA3_512} hash,
      BLOB key,
      BLOB msg
   )
Geoff Simmons's avatar
Geoff Simmons committed
506 507 508 509 510 511 512 513 514 515 516 517 518 519 520

Returns the HMAC for ``msg`` as specified by ``hash`` and the ``key``.

As with the ``digest`` object and ``hash`` function, the use of the
``hmac`` object is likely to be more efficient than the ``hmacf``
function, because the internal cryptographic state of the HMAC for a
given key is pre-computed in the object constructor. So if the key is
fixed and known at initialization time, then you should use the
``hmac`` object. The ``hmacf`` function should only be used if the key
is not known at runtime, or, for example, should be changed without
requiring a VCL reload.

Example::

	import blobdigest;
521
	import blob;
Geoff Simmons's avatar
Geoff Simmons committed
522 523 524 525 526

	# Decode the base64-encoded string as a HMAC key, compute the
	# SHA512 HMAC of the Msg header, and save the hex-encoded
	# result in a header.
	set req.http.HMAC
527
	    = blob.encode(HEX, UPPER,
Geoff Simmons's avatar
Geoff Simmons committed
528
	                      blobdigest.hmacf(SHA512,
Nils Goroll's avatar
Nils Goroll committed
529 530
			                       blob.decode(decoding=BASE64, encoded="Zm9v"),
					       blob.decode(encoded=
Geoff Simmons's avatar
Geoff Simmons committed
531 532
					                       req.http.Msg)));

533

534
.. _func_version:
535

536 537
STRING version()
----------------
538

539 540 541 542 543 544 545 546 547
Returns the version string for this VMOD.

Example::

	std.log("Using VMOD blobdigest version " + blobdigest.version());

REQUIREMENTS
============

Nils Goroll's avatar
Nils Goroll committed
548
This version of the VMOD requires Varnish since version 5.2. See the
549 550
source repository for versions that are compatible with other versions
of Varnish.
551

Nils Goroll's avatar
Nils Goroll committed
552
The gcc compiler and Perl 5 are required for the build.
553

554 555 556
LIMITATIONS
===========

557 558
For operations outside of ``vcl_init``, the VMOD allocates memory for
BLOBs and other internal structures in Varnish workspace.  If its
559 560 561 562 563
methods or functions fail, as indicated by "out of space" messages in
the Varnish log (with the ``VCL_Error`` tag), then you will need to
increase the varnishd parameters ``workspace_client`` and/or
``workspace_backend``.

564 565 566
For operations invoked in ``vcl_init``, the VMOD allocates heap
memory, and hence is only limited by available RAM.

567 568 569 570 571
INSTALLATION
============

See `INSTALL.rst <INSTALL.rst>`_ in the source repository.

572 573 574 575 576 577 578 579 580 581
AUTHOR
======

* Geoffrey Simmons <geoff@uplex.de>

UPLEX Nils Goroll Systemoptimierung

Cryptographic code is adapted from librhash by Aleksey Kravchenko,
which is in the public domain (see `LICENSE <LICENSE>`_).

Geoff Simmons's avatar
Geoff Simmons committed
582 583 584 585 586 587 588
CONTRIBUTING
============

See `CONTRIBUTING.rst <CONTRIBUTING.rst>`_ in the source repository
for notes on contributing source code and documentation, raising
issues, and for developer guidelines.

589 590
SEE ALSO
========
591

592 593
* varnishd(1)
* vcl(7)
594
* source repository: https://code.uplex.de/uplex-varnish/libvmod-blobdigest
595
* RHash:
596

597 598 599
  * https://github.com/rhash/RHash
  * http://rhash.anz.ru/

600

601 602 603
COPYRIGHT
=========

604 605 606 607 608 609 610 611 612
::

  Copyright (c) 2016 UPLEX Nils Goroll Systemoptimierung
  All rights reserved
 
  Author: Geoffrey Simmons <geoffrey.simmons@uplex.de>
 
  See LICENSE