Basic zipflow implementation (POC)

limited to a single file for now.

COPYING

+See LICENSE for details.
+

LICENSE

+Copyright 2022 UPLEX Nils Goroll Systemoptimierung
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions
+are met:
+1. Redistributions of source code must retain the above copyright
+   notice, this list of conditions and the following disclaimer.
+2. Redistributions in binary form must reproduce the above copyright
+   notice, this list of conditions and the following disclaimer in the
+   documentation and/or other materials provided with the distribution.
+
+THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGE.

README.rst

+..
+.. NB:  This file is machine generated, DO NOT EDIT!
+..
+.. Edit ./vmod_zipflow.vcc and run make instead
+..
+
+.. role:: ref(emphasis)
+
+============
+vmod_zipflow
+============
+
+----------------------
+Varnish zipflow Module
+----------------------
+
+:Manual section: 3
+
+SYNOPSIS
+========
+
+.. parsed-literal::
+
+  import zipflow [as name] [from "path"]
+  
+  VOID set_level(INT level)
+  
+  VOID meta(STRING name, STRING mode, TIME atime, TIME mtime)
+  
+NOTE
+====
+
+This vmod needs https://github.com/varnishcache/varnish-cache/pull/3843
+
+DESCRIPTION
+===========
+
+.. _zipflow: https://github.com/madler/zipflow
+
+This module provides a Varnish Delivery Processor (VDP) interface to
+Mark Adler's `zipflow`_ library to package and compress
+responses into the ZIP format.
+
+Example
+    ::
+
+	import zipflow;
+
+	sub vcl_init {
+		zipflow.set_level(9);
+	}
+
+	sub vcl_deliver {
+		zipflow.meta("filename");
+		set resp.filters += " zipflow";
+	}
+
+.. _zipflow.set_level():
+
+VOID set_level(INT level)
+-------------------------
+
+Set the (default) compression level for the zipflow filter.
+
+Valid values for `level` are -1 to select the default compression
+level ``Z_DEFAULT_COMPRESSION``, which is also the overall default, or
+between 0 and 9: 1 gives best speed, 9 gives best compression, 0 gives
+no compression at all (the input data is simply copied a block at a
+time).  ``Z_DEFAULT_COMPRESSION`` requests a default compromise
+between speed and compression (currently equivalent to level 6). [Text
+copied from zlib.h]
+
+When used from ``vcl_init {}``, sets the default for all tasks from
+all VCLs. Any such call affects all VCLs using this vmod, setting the
+default per VCL is not currently supported.
+
+When used elsewhere, sets the level for the current task. Calls from
+``esi_level`` > 0 have no effect.
+
+.. _zipflow.meta():
+
+VOID meta(STRING name, STRING mode, TIME atime, TIME mtime)
+-----------------------------------------------------------
+
+::
+
+   VOID meta(
+      STRING name,
+      STRING mode="0644",
+      TIME atime=0,
+      TIME mtime=0
+   )
+
+Set zip metadata for the currently streamed object:
+
+* `name`: Filename
+
+* `mode`: UNIX mode bits, default to 0644
+
+  Note: This argument is of string type and interpreted as *octal*.
+
+* `atime`: Access time, defaults to ``now``.
+
+* `mtime`: Modification time, defaults to ``now``.
+
+SEE ALSO
+========
+
+vcl\(7),
+varnishd\(1)
+
+COPYRIGHT
+=========
+
+::
+
+  Copyright 2022 UPLEX Nils Goroll Systemoptimierung
+  All rights reserved
+ 
+  Author: Nils Goroll <nils.goroll@uplex.de>
+ 
+  See LICENSE
+ 

zipflow @ 95ae59fc

-Subproject commit 913ef458c51504d1f461f6d3a24d4a1a9bdc2bcc
+Subproject commit 95ae59fc13f684a19dc2f4875eaba4410564c2a6

src/Makefile.am

@@ -21,7 +21,7 @@ AM_TESTS_ENVIRONMENT = \
 	PATH="$(abs_builddir):$(VARNISH_TEST_PATH):$(PATH)" \
 	LD_LIBRARY_PATH="$(VARNISH_LIBRARY_PATH)"
 TEST_EXTENSIONS = .vtc
-VTC_LOG_COMPILER = varnishtest -v
+VTC_LOG_COMPILER = varnishtest -vl -t10
 AM_VTC_LOG_FLAGS = \
 	-p vcl_path="$(abs_top_srcdir)/vcl:$(VARNISHAPI_VCLDIR)" \
 	-p vmod_path="$(abs_builddir)/.libs:$(vmoddir):$(VARNISHAPI_VMODDIR)"

src/vmod_zipflow.c

+/*-
+ * Copyright 2022 UPLEX Nils Goroll Systemoptimierung
+ * All rights reserved
+ *
+ * Author: Nils Goroll <nils.goroll@uplex.de>
+ *
+ * Redistribution and use in source and binary forms, with or without
+ * modification, are permitted provided that the following conditions
+ * are met:
+ * 1. Redistributions of source code must retain the above copyright
+ *    notice, this list of conditions and the following disclaimer.
+ * 2. Redistributions in binary form must reproduce the above copyright
+ *    notice, this list of conditions and the following disclaimer in the
+ *    documentation and/or other materials provided with the distribution.
+ *
+ * THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
+ * ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+ * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+ * ARE DISCLAIMED.  IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
+ * FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+ * DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+ * OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+ * HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+ * LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+ * OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+ * SUCH DAMAGE.
+ */
+
 #include "config.h"
 
+#include <string.h>	// memset() for INIT_OBJ()
+#include <stdlib.h>	// strtoul()
+
 #include <cache/cache.h>
+#include <cache/cache_filter.h>
+#include <vcl.h>
 
 #include "vcc_zipflow_if.h"
 
-VCL_STRING
-vmod_hello(VRT_CTX)
+#include "zlib.h"
+#include "zipflow.h"
+
+static void __attribute__((constructor))
+assert_zlib(void)
+{
+	assert(Z_DEFAULT_COMPRESSION == -1);
+}
+
+// https://github.com/varnishcache/varnish-cache/pull/3815
+#ifndef WS_TASK_ALLOC_OBJ
+#define WS_TASK_ALLOC_OBJ(ctx, ptr, magic) do {			\
+	ptr = WS_Alloc((ctx)->ws, sizeof *(ptr));		\
+	if ((ptr) == NULL)					\
+		VRT_fail(ctx, "Out of workspace for " #magic);	\
+	else							\
+		INIT_OBJ(ptr, magic);				\
+} while(0)
+#endif
+
+static int default_level = Z_DEFAULT_COMPRESSION;
+
+static const void *zipflow_priv = &zipflow_priv;
+
+struct zipflow_task {
+	unsigned		magic;
+#define ZIPFLOW_TASK_MAGIC	0xaa175160
+	int			level;
+	VCL_STRING		name;
+	unsigned		mode;
+	uint32_t		atime, mtime;
+	ZIP			*zip;
+};
+
+static struct vdp vdp_zipflow;
+
+static struct zipflow_task *
+get_zipflow_task(VRT_CTX)
+{
+	struct vmod_priv *task_priv;
+	struct zipflow_task *zft;
+
+	task_priv = VRT_priv_task(ctx, zipflow_priv);
+	if (task_priv == NULL) {
+		VRT_fail(ctx, "Can not get PRIV_TASK");
+		return (NULL);
+	}
+
+	if (task_priv->priv != NULL) {
+		CAST_OBJ_NOTNULL(zft, task_priv->priv, ZIPFLOW_TASK_MAGIC);
+		return (zft);
+	}
+
+	WS_TASK_ALLOC_OBJ(ctx, zft, ZIPFLOW_TASK_MAGIC);
+	if (zft == NULL)
+		return (NULL);
+
+	zft->level = default_level;
+	zft->name = "unnamed_file";
+	zft->mode = 0644;
+	zft->atime = ctx->now;
+	zft->mtime = ctx->now;
+	return (zft);
+}
+
+/*
+ * ============================================================
+ * VCL interface
+ */
+
+VCL_VOID
+vmod_set_level(VRT_CTX, VCL_INT level)
 {
+	struct zipflow_task *zft;
 
 	CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
-	return ("vmod-zipflow");
+
+	if (level < -1 || level > 9) {
+		VRT_fail(ctx, "invalid level %ld", level);
+		return;
+	}
+
+	if (ctx->method == VCL_MET_INIT) {
+		default_level = level;
+		return;
+	}
+	if (ctx->method & VCL_MET_TASK_H) {
+		VRT_fail(ctx, "set_level can not be called here");
+		return;
+	}
+
+	AN(ctx->method & (VCL_MET_TASK_C | VCL_MET_TASK_B));
+	zft = get_zipflow_task(ctx);
+	if (zft == NULL)
+		return;
+	zft->level = level;
+}
+
+VCL_VOID
+vmod_meta(VRT_CTX, VCL_STRING name, VCL_STRING mode,
+    VCL_TIME atime, VCL_TIME mtime)
+{
+	unsigned long u;
+	struct zipflow_task *zft = get_zipflow_task(ctx);
+
+	if (zft == NULL)
+		return;
+
+	u = strtoul(mode, NULL, 8);
+	if (u == ULONG_MAX) {
+		VRT_fail(ctx, ".meta() error converting mode: %d (%s)",
+		    errno, VAS_errtxt(errno));
+		return;
+	}
+	if (atime == 0)
+		atime = ctx->now;
+	if (mtime == 0)
+		mtime = ctx->now;
+
+	zft->name = name;
+	zft->mode = u;
+	zft->atime = atime;
+	zft->mtime = mtime;
+}
+
+int
+vmod_event(VRT_CTX, struct vmod_priv *priv, enum vcl_event_e e)
+{
+	(void) priv;
+
+	switch(e) {
+	case VCL_EVENT_LOAD:
+		if (VRT_AddFilter(ctx, NULL, &vdp_zipflow) != NULL)
+			return (1);
+		break;
+	case VCL_EVENT_DISCARD:
+		VRT_RemoveFilter(ctx, NULL, &vdp_zipflow);
+		break;
+	case VCL_EVENT_WARM:
+	case VCL_EVENT_COLD:
+		break;
+	default:
+		WRONG("illegal event enum");
+	}
+	return (0);
 }
+
+/*
+ * ============================================================
+ * VDP
+ */
+
+
+static int
+vdp_zipflow_put(void *priv, void const *ptr, size_t len)
+{
+
+	if (ptr == NULL || len == 0)
+		return (0);
+
+	return (VDP_bytes(priv, VDP_FLUSH, ptr, len));
+}
+
+static void
+vdp_zipflow_log(void *vsl, char *msg)
+{
+
+	AN(vsl);
+	AN(msg);
+	VSLb(vsl, SLT_Error, "zipflow: %s", msg);
+	free(msg);
+}
+
+static int
+vdp_zipflow_init(VRT_CTX, struct vdp_ctx *vdc, void **priv, struct objcore *oc)
+{
+	struct zipflow_task *zft = get_zipflow_task(ctx);
+	struct req *req;
+
+	if (zft == NULL)
+		return (1);
+
+	AN(zft->name);
+
+	AZ(zft->zip);
+	zft->zip = zip_pipe(vdc, vdp_zipflow_put, zft->level);
+	if (zft->zip == NULL) {
+		VSLb(vdc->vsl, SLT_Error, "zipflow: zip_pipe failed");
+		return (1);
+	}
+	AZ(zip_log(zft->zip, vdc->vsl, vdp_zipflow_log));
+	AZ(zip_meta(zft->zip, zft->name, 3, zft->mode, zft->atime, zft->mtime));
+
+	AN(priv);
+	AZ(*priv);
+	*priv = zft;
+
+	req = vdc->req;
+	CHECK_OBJ_NOTNULL(req, REQ_MAGIC);
+	RFC2616_Weaken_Etag(req->resp);
+	if (req->resp_len != 0)
+		req->resp_len = -1;
+
+	return (0);
+}
+
+static int
+vdp_zipflow_fini(struct vdp_ctx *vdc, void **priv)
+{
+	struct zipflow_task *zft;
+	int r;
+
+	AN(priv);
+	zft = *priv;
+	*priv = NULL;
+	if (zft != NULL) {
+		CHECK_OBJ(zft, ZIPFLOW_TASK_MAGIC);
+		r = zip_close(zft->zip);
+		if (r)
+			VSLb(vdc->vsl, SLT_Error, "zip_close returned %d", r);
+		memset(zft, 0, sizeof *zft);
+	}
+	return (0);
+}
+
+static int
+vdp_zipflow_bytes(struct vdp_ctx *vdc, enum vdp_action act, void **priv,
+    const void *ptr, ssize_t len)
+{
+	struct zipflow_task *zft;
+	int r;
+
+	AN(priv);
+	CAST_OBJ_NOTNULL(zft, *priv, ZIPFLOW_TASK_MAGIC);
+	r = zip_data(zft->zip, ptr, len, act == VDP_END);
+	if (r) {
+		VSLb(vdc->vsl, SLT_Error, "zip_data returned %d", r);
+		return (1);
+	}
+
+	if (act != VDP_END)
+		return (0);
+
+	*priv = NULL;
+	r = zip_close(zft->zip);
+	if (r)
+		VSLb(vdc->vsl, SLT_Error, "zip_close returned %d", r);
+	memset(zft, 0, sizeof *zft);
+	return (VDP_bytes(vdc, VDP_END, NULL, 0));
+}
+
+static struct vdp vdp_zipflow = {
+	.name = "zipflow",
+	.init = vdp_zipflow_init,
+	.bytes = vdp_zipflow_bytes,
+	.fini = vdp_zipflow_fini
+};

src/vmod_zipflow.vcc

+#-
+# Copyright 2022 UPLEX Nils Goroll Systemoptimierung
+# All rights reserved
+#
+# Author: Nils Goroll <nils.goroll@uplex.de>
+#
+# See LICENSE
+#
+
 $Module zipflow 3 "Varnish zipflow Module"
 
+NOTE
+====
+
+This vmod needs https://github.com/varnishcache/varnish-cache/pull/3843
+
 DESCRIPTION
 ===========
 
-This VCC file was generated by VCDK, it is used to for both the VMOD
-interface and its manual using reStructuredText.
+.. _zipflow: https://github.com/madler/zipflow
 
-XXX: document vmod-zipflow
+This module provides a Varnish Delivery Processor (VDP) interface to
+Mark Adler's `zipflow`_ library to package and compress
+responses into the ZIP format.
 
 Example
     ::
 
-        import zipflow;
+	import zipflow;
+
+	sub vcl_init {
+		zipflow.set_level(9);
+	}
+
+	sub vcl_deliver {
+		zipflow.meta("filename");
+		set resp.filters += " zipflow";
+	}
+
+$Function VOID set_level(INT level)
+
+Set the (default) compression level for the zipflow filter.
 
-        sub vcl_deliver {
-	    set resp.http.Hello = zipflow.hello();
-        }
+Valid values for `level` are -1 to select the default compression
+level ``Z_DEFAULT_COMPRESSION``, which is also the overall default, or
+between 0 and 9: 1 gives best speed, 9 gives best compression, 0 gives
+no compression at all (the input data is simply copied a block at a
+time).  ``Z_DEFAULT_COMPRESSION`` requests a default compromise
+between speed and compression (currently equivalent to level 6). [Text
+copied from zlib.h]
 
-XXX: define vmod-zipflow interface
+When used from ``vcl_init {}``, sets the default for all tasks from
+all VCLs. Any such call affects all VCLs using this vmod, setting the
+default per VCL is not currently supported.
 
-$Function STRING hello()
+When used elsewhere, sets the level for the current task. Calls from
+``esi_level`` > 0 have no effect.
 
-Description
-    Hello world for vmod-zipflow
+$Function VOID meta(STRING name, STRING mode="0644", TIME atime=0, TIME mtime=0)
+
+Set zip metadata for the currently streamed object:
+
+* `name`: Filename
+
+* `mode`: UNIX mode bits, default to 0644
+
+  Note: This argument is of string type and interpreted as *octal*.
+
+* `atime`: Access time, defaults to ``now``.
+
+* `mtime`: Modification time, defaults to ``now``.
 
 SEE ALSO
 ========
 
 vcl\(7),
 varnishd\(1)
+
+$Event event

src/vtc/vmod_zipflow.vtc

@@ -2,20 +2,27 @@ varnishtest "test vmod-zipflow"
 
 server s1 {
 	rxreq
-	txresp
+	txresp -body "zip file content"
 } -start
 
 varnish v1 -vcl+backend {
-        import zipflow;
+	import zipflow;
 
-        sub vcl_deliver {
-	    set resp.http.Hello = zipflow.hello();
-        }
+	sub vcl_init {
+		zipflow.set_level(1);
+	}
+
+	sub vcl_deliver {
+		zipflow.set_level(-1);
+		zipflow.meta("filename", mode="444", atime=now, mtime=now - 1d);
+		set resp.filters += " zipflow";
+	}
 } -start
 
 client c1 {
 	txreq
 	rxresp
 	expect resp.status == 200
-	expect resp.http.Hello == "vmod-zipflow"
 } -run
+
+shell "curl -svo t.zip -H 'Host: ${v1_addr}' http://${v1_addr}:${v1_port}/ && unzip -l t.zip"