Add the fileread() function.

README.rst

@@ -175,6 +175,7 @@ CONTENTS
 
 * VOID init(ENUM {INIT_SECMEM,DISABLE_SECMEM,FINISH}, BYTES)
 * symmetric(ENUM {AES,AES128,RIJNDAEL,RIJNDAEL128,AES192,RIJNDAEL192,AES256,RIJNDAEL256}, ENUM {ECB,CFB,CBC,OFB,CTR}, ENUM {PKCS7,ISO7816,X923,NONE}, BLOB, BOOL, BOOL)
+* BLOB fileread(PRIV_TASK, STRING)
 * BLOB random(PRIV_TASK, ENUM {STRONG,VERY_STRONG,NONCE}, BYTES)
 * INT random_int(ENUM {STRONG,NONCE}, INT)
 * VOID wipe(BLOB)
@@ -511,6 +512,99 @@ Examples::
                          iv=blobcode.decode(BASE64, req.http.X-IV-256)));
   }
 
+.. _func_fileread:
+
+fileread
+--------
+
+::
+
+	BLOB fileread(PRIV_TASK, STRING path)
+
+Return the contents of the file at ``path`` in a BLOB. This function
+is provided for the specific purpose of reading sensitive data that
+are needed during ``vcl_init``, such as cryptographic keys, so that
+they are not exposed in the VCL source.
+
+If secure memory is enabled, then space for the BLOB contents is
+allocated from the secure memory pool; otherwise, space is allocated
+from the heap. The contents are retained in memory for the duration
+of the current task scope; this means:
+
+* If ``fileread()`` is called in ``vcl_init`` (or ``vcl_fini``), then
+  the scope ends when the execution of that subroutine ends.
+
+* Otherwise (in any other VCL subroutine), the scope ends when the
+  current client or backend transaction ends.
+
+For example, if ``fileread()`` is called during one of the
+``vcl_backend_*`` subroutines, then the scope ends when the current
+backend transaction is complete.
+
+When the task scope ends, then the BLOB contents are overwritten in
+memory, and the allocated memory is freed. If the contents were
+allocated from the secure memory pool, then that space is returned to
+the pool.
+
+``fileread()`` fails and returns NULL if:
+
+* libgcrypt initialization is not finished
+
+* ``path`` is NULL
+
+* ``path`` does not denote a regular file. For example, ``path`` MAY
+  NOT be a directory, symbolic link or named pipe.
+
+* ``path`` cannot be inspected by stat(2), opened, read or closed,
+  for example if the file does not exist, or if the Varnish child
+  process has insufficient privileges to read it.
+
+* ``path`` is changed at some time between just after invocation of
+  ``fileread()`` and just after reading its contents.
+
+* There is insufficient space for the necessary allocations, for
+  example if the secure memory pool is to be used but does not have
+  enough free memory.
+
+If ``fileread()`` fails, it writes an error message to the Varnish log
+with the tag ``VCL_Error``. If it fails during ``vcl_init``, then the
+VCL load fails with the error message.
+
+``fileread()`` does not do any binary-to-text conversion, so the
+contents of the file should be the raw binary data that you intend to
+read into the BLOB. The file should not, for example, contain a hex or
+base64 encoding.
+
+If you intend to read sensitive data from a file, consider setting the
+file's permissions so that only the owner of the Varnish child process
+can read it. Secure management of the file becomes part of your
+administrative responsibility that is outside the scope of Varnish and
+this VMOD.
+
+``fileread()`` can be used to read data from a file in any VCL
+subroutine, and the use case does not have to be related to
+cryptography, but it is very inefficient. It reads from file and
+allocates memory on every invocation, and that memory is wiped and
+freed at the end of every task scope in which it is invoked, all of
+which can have a severe impact on performance. See the ``fileread()``
+funciton of the std VMOD (vmod_std(3)) for an efficient way to read
+from files.
+
+Example::
+
+  # Write 16 bytes of binary data to a file.
+  $ echo 'AAECAwQFBgcICQoLDA0ODw==' | base64 -d > /path/to/key
+
+  # Make the file readable only to the Varnish worker process owner.
+  $ chown vcache:varnish /path/to/key
+  $ chmod 400 /path/to/key
+
+  # In VCL, use the file contents as an encryption key.
+  sub vcl_init {
+      new aes128 = gcrypt.symmetric(AES128, CTR,
+                                    key=gcrypt.fileread("/path/to/key"));
+  }
+
 .. _func_random:
 
 random

src/tests/fileread.vtc

+# looks like -*- vcl -*-
+
+varnishtest "fileread()"
+
+shell {printf "0123456789abcdef" > ${tmpdir}/f.txt}
+shell {rm -f ${tmpdir}/nonexistent.txt}
+shell {mkfifo ${tmpdir}/fifo.txt}
+
+varnish v1 -vcl { backend b { .host = "${bad_ip}"; } } -start
+
+varnish v1 -errvcl {vmod gcrypt error: libgcrypt initialization not finished in gcrypt.fileread()} {
+	import gcrypt from "${vmod_topbuild}/src/.libs/libvmod_gcrypt.so";
+	backend b { .host = "${bad_ip}"; }
+
+	sub vcl_init {
+		new aes = gcrypt.symmetric(AES, ECB, NONE,
+		              key=gcrypt.fileread("${tmpdir}/nonexistent.txt"));
+	}
+}
+
+varnish v1 -vcl {
+	import blobcode;
+	import gcrypt from "${vmod_topbuild}/src/.libs/libvmod_gcrypt.so";
+	backend b { .host = "${bad_ip}"; }
+
+	sub vcl_init {
+		gcrypt.init(FINISH);
+	}
+
+	sub vcl_recv {
+		return(synth(200));
+	}
+
+	sub vcl_synth {
+		set resp.http.f1 = blobcode.encode(IDENTITY,
+		                            gcrypt.fileread("${tmpdir}/f.txt"));
+		set resp.http.f2 = blobcode.encode(IDENTITY,
+		                            gcrypt.fileread("${tmpdir}/f.txt"));
+		set resp.http.n = blobcode.encode(IDENTITY,
+		                  gcrypt.fileread("${tmpdir}/nonexistent.txt"));
+		set resp.http.fifo = blobcode.encode(IDENTITY,
+		                         gcrypt.fileread("${tmpdir}/fifo.txt"));
+
+		return(deliver);
+	}
+}
+
+client c1 {
+	txreq
+	rxresp
+	expect resp.status == 200
+	expect resp.http.f1 == "0123456789abcdef"
+	expect resp.http.f2 == "0123456789abcdef"
+	expect resp.http.n == ""
+	expect resp.http.fifo == ""
+} -run
+
+logexpect l1 -v v1 -d 1 -q "VCL_Error" {
+	expect 0 * Begin req
+	expect * = VCL_Error "^vmod gcrypt error: Cannot stat .+nonexistent.txt in gcrypt.fileread..: .+$"
+	expect * = VCL_Error "^vmod gcrypt error: .+fifo.txt is not a regular file in gcrypt.fileread..$"
+	expect * = End
+} -run
+
+# The intended use case
+varnish v1 -vcl {
+	import gcrypt from "${vmod_topbuild}/src/.libs/libvmod_gcrypt.so";
+	backend b { .host = "${bad_ip}"; }
+
+	sub vcl_init {
+		new aes = gcrypt.symmetric(AES, ECB, NONE,
+		                        key=gcrypt.fileread("${tmpdir}/f.txt"));
+	}
+}
+
+varnish v1 -errvcl {vmod gcrypt error: Cannot stat} {
+	import gcrypt from "${vmod_topbuild}/src/.libs/libvmod_gcrypt.so";
+	backend b { .host = "${bad_ip}"; }
+
+	sub vcl_init {
+		new aes = gcrypt.symmetric(AES, ECB, NONE,
+		              key=gcrypt.fileread("${tmpdir}/nonexistent.txt"));
+	}
+}
+
+varnish v1 -errvcl {is not a regular file} {
+	import gcrypt from "${vmod_topbuild}/src/.libs/libvmod_gcrypt.so";
+	backend b { .host = "${bad_ip}"; }
+
+	sub vcl_init {
+		new aes = gcrypt.symmetric(AES, ECB, NONE,
+		                     key=gcrypt.fileread("${tmpdir}/fifo.txt"));
+	}
+}

src/vmod_gcrypt.c

@@ -35,12 +35,17 @@
 #include <pthread.h>
 #include <errno.h>
 #include <stdint.h>
+#include <sys/types.h>
+#include <sys/stat.h>
+#include <unistd.h>
+#include <fcntl.h>
 
 #include "vcl.h"
 #include "cache/cache.h"
 #include "vrt.h"
 #include "vas.h"
 #include "vdef.h"
+#include "vqueue.h"
 
 #include "vcc_if.h"
 #include "vmod_gcrypt.h"
@@ -76,6 +81,16 @@ struct vmod_gcrypt_symmetric {
 	unsigned int		flags;
 };
 
+struct filedata {
+	unsigned		magic;
+#define VMOD_GCRYPT_FILEDATA_MAGIC 0xb6250b0e
+	VSLIST_ENTRY(filedata)	list;
+	void			*contents;
+	size_t			len;
+};
+
+VSLIST_HEAD(filedata_head, filedata);
+
 static const char *gcrypt_version = NULL;
 static int secmem_enabled = 1;
 
@@ -746,6 +761,8 @@ vmod_random_int(VRT_CTX, VCL_ENUM qualitys, VCL_INT bound)
 	return r;
 }
 
+/* Function wipe */
+
 static inline void
 wipe(void * const dst, size_t len, uint8_t val)
 {
@@ -784,6 +801,179 @@ vmod_wipe(VRT_CTX, VCL_BLOB b)
 	wipe(b->priv, b->len, 0x00);
 }
 
+/* Function fileread */
+
+static void
+filedata_free(void *p)
+{
+	struct filedata_head *fhead;
+	struct filedata *f;
+
+	if (p == NULL)
+		return;
+	fhead = p;
+	f = VSLIST_FIRST(fhead);
+	while (f != NULL) {
+		struct filedata *next;
+
+		CHECK_OBJ(f, VMOD_GCRYPT_FILEDATA_MAGIC);
+		if (f->contents != NULL) {
+			wipe(f->contents, f->len, 0xff);
+			wipe(f->contents, f->len, 0xaa);
+			wipe(f->contents, f->len, 0x55);
+			wipe(f->contents, f->len, 0x00);
+			if (secmem_enabled)
+				gcry_free(f->contents);
+			else
+				free(f->contents);
+		}
+		next = VSLIST_NEXT(f, list);
+		FREE_OBJ(f);
+		f = next;
+	}
+	free(fhead);
+}
+
+VCL_BLOB
+vmod_fileread(VRT_CTX, struct vmod_priv *task, VCL_STRING path)
+{
+	struct vmod_priv *b;
+	struct filedata_head *fhead;
+	struct filedata *fdata;
+	struct stat st, fst;
+	uintptr_t snap;
+	void *contents = NULL;
+	int fd = -1;
+
+	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
+	AN(task);
+	if (path == NULL) {
+		ERR(ctx, "path is NULL in gcrypt.fileread()");
+		return NULL;
+	}
+	if (!gcry_control(GCRYCTL_INITIALIZATION_FINISHED_P)) {
+		ERR(ctx, "libgcrypt initialization not finished in "
+		    "gcrypt.fileread()");
+		return NULL;
+	}
+
+	if (task->priv == NULL) {
+		fhead = calloc(1, sizeof(*fhead));
+		if (fhead == NULL) {
+			ERRNOMEM(ctx, "Cannot allocate file data list in "
+				 "gcrypt.fileread()");
+			return NULL;
+		}
+		VSLIST_INIT(fhead);
+		task->priv = fhead;
+		task->len = sizeof(*fhead);
+		task->free = filedata_free;
+	}
+	else
+		fhead = task->priv;
+
+	snap = WS_Snapshot(ctx->ws);
+	if ((b = WS_Alloc(ctx->ws, sizeof(*b))) == NULL) {
+		ERRNOMEM(ctx, "Allocating return BLOB in gcrypt.fileread()");
+		return NULL;
+	}
+	b->free = NULL;
+
+	errno = 0;
+	if (stat(path, &st) < 0) {
+		VERR(ctx, "Cannot stat %s in gcrypt.fileread(): %s", path,
+		     strerror(errno));
+		goto fail;
+	}
+
+	if (!S_ISREG(st.st_mode)) {
+		VERR(ctx, "%s is not a regular file in gcrypt.fileread()",
+		     path);
+		goto fail;
+	}
+	errno = 0;
+	if ((fd = open(path, O_RDONLY)) < 0) {
+		VERR(ctx, "Cannot open %s in gcrypt.fileread(): %s", path,
+		     strerror(errno));
+		goto fail;
+	}
+
+	if (secmem_enabled)
+		contents = gcry_malloc_secure(st.st_size);
+	else
+		contents = malloc(st.st_size);
+	if (contents == NULL) {
+		VERRNOMEM(ctx, "Allocating space for contents of %s in "
+			  "gcrypt.fileread()", path);
+		goto fail;
+	}
+	for (size_t len = st.st_size, offset = 0; len > 0; ) {
+		ssize_t bytes;
+
+		errno = 0;
+		bytes = read(fd, contents + offset, len);
+		if (bytes < 0) {
+			VERR(ctx, "Reading from %s in gcrypt.fileread(): %s",
+			     path, strerror(errno));
+			goto fail;
+		}
+		len -= bytes;
+		offset += bytes;
+	}
+
+	/* TOCTOU check */
+	errno = 0;
+	if (fstat(fd, &fst) < 0) {
+		VERR(ctx, "Cannot stat %s after read in gcrypt.fileread(): %s",
+		     path, strerror(errno));
+		goto fail;
+	}
+	if (fst.st_mtime != st.st_mtime || fst.st_mode != st.st_mode
+	    || fst.st_size != st.st_size || fst.st_uid != st.st_uid
+	    || fst.st_gid != st.st_gid || fst.st_dev != st.st_dev
+	    || fst.st_ino != st.st_ino) {
+		VERR(ctx, "%s was changed between stat and read", path);
+		goto fail;
+	}
+
+	errno = 0;
+	if (close(fd) < 0) {
+		VERR(ctx, "Closing %s after read in gcrypt.fileread(): %s",
+		     path, strerror(errno));
+		fd = -1;
+		goto fail;
+	}
+	fd = -1;
+
+	ALLOC_OBJ(fdata, VMOD_GCRYPT_FILEDATA_MAGIC);
+	if (fdata == NULL) {
+		ERRNOMEM(ctx, "Saving file data in gcrypt.fileread()");
+		goto fail;
+	}
+	fdata->contents = contents;
+	fdata->len = st.st_size;
+	VSLIST_INSERT_HEAD(fhead, fdata, list);
+	b->priv = contents;
+	b->len = st.st_size;
+	return b;
+
+ fail:
+	if (contents != NULL) {
+		if (secmem_enabled)
+			gcry_free(contents);
+		else
+			free(contents);
+	}
+	if (fd != -1) {
+		errno = 0;
+		if (close(fd) < 0)
+			VERR(ctx, "Closing %s in gcrypt.fileread(): %s", path,
+			     strerror(errno));
+	}
+	WS_Reset(ctx->ws, snap);
+	return NULL;
+}
+
 VCL_STRING
 vmod_version(VRT_CTX __attribute__((unused)))
 {

src/vmod_gcrypt.vcc

@@ -458,6 +458,92 @@ Examples::
                          iv=blobcode.decode(BASE64, req.http.X-IV-256)));
   }
 
+$Function BLOB fileread(PRIV_TASK, STRING path)
+
+Return the contents of the file at ``path`` in a BLOB. This function
+is provided for the specific purpose of reading sensitive data that
+are needed during ``vcl_init``, such as cryptographic keys, so that
+they are not exposed in the VCL source.
+
+If secure memory is enabled, then space for the BLOB contents is
+allocated from the secure memory pool; otherwise, space is allocated
+from the heap. The contents are retained in memory for the duration
+of the current task scope; this means:
+
+* If ``fileread()`` is called in ``vcl_init`` (or ``vcl_fini``), then
+  the scope ends when the execution of that subroutine ends.
+
+* Otherwise (in any other VCL subroutine), the scope ends when the
+  current client or backend transaction ends.
+
+For example, if ``fileread()`` is called during one of the
+``vcl_backend_*`` subroutines, then the scope ends when the current
+backend transaction is complete.
+
+When the task scope ends, then the BLOB contents are overwritten in
+memory, and the allocated memory is freed. If the contents were
+allocated from the secure memory pool, then that space is returned to
+the pool.
+
+``fileread()`` fails and returns NULL if:
+
+* libgcrypt initialization is not finished
+
+* ``path`` is NULL
+
+* ``path`` does not denote a regular file. For example, ``path`` MAY
+  NOT be a directory, symbolic link or named pipe.
+
+* ``path`` cannot be inspected by stat(2), opened, read or closed,
+  for example if the file does not exist, or if the Varnish child
+  process has insufficient privileges to read it.
+
+* ``path`` is changed at some time between just after invocation of
+  ``fileread()`` and just after reading its contents.
+
+* There is insufficient space for the necessary allocations, for
+  example if the secure memory pool is to be used but does not have
+  enough free memory.
+
+If ``fileread()`` fails, it writes an error message to the Varnish log
+with the tag ``VCL_Error``. If it fails during ``vcl_init``, then the
+VCL load fails with the error message.
+
+``fileread()`` does not do any binary-to-text conversion, so the
+contents of the file should be the raw binary data that you intend to
+read into the BLOB. The file should not, for example, contain a hex or
+base64 encoding.
+
+If you intend to read sensitive data from a file, consider setting the
+file's permissions so that only the owner of the Varnish child process
+can read it. Secure management of the file becomes part of your
+administrative responsibility that is outside the scope of Varnish and
+this VMOD.
+
+``fileread()`` can be used to read data from a file in any VCL
+subroutine, and the use case does not have to be related to
+cryptography, but it is very inefficient. It reads from file and
+allocates memory on every invocation, and that memory is wiped and
+freed at the end of every task scope in which it is invoked, all of
+which can have a severe impact on performance. See the ``fileread()``
+funciton of the std VMOD (vmod_std(3)) for an efficient way to read
+from files.
+
+Example::
+
+  # Write 16 bytes of binary data to a file.
+  $ echo 'AAECAwQFBgcICQoLDA0ODw==' | base64 -d > /path/to/key
+
+  # Make the file readable only to the Varnish worker process owner.
+  $ chown vcache:varnish /path/to/key
+  $ chmod 400 /path/to/key
+
+  # In VCL, use the file contents as an encryption key.
+  sub vcl_init {
+      new aes128 = gcrypt.symmetric(AES128, CTR,
+                                    key=gcrypt.fileread("/path/to/key"));
+  }
+
 $Function BLOB random(PRIV_TASK, ENUM {STRONG, VERY_STRONG, NONCE} quality=0,
 		      BYTES n=0)