Initial public release

parents
# build system
.deps/
.libs/
autom4te.cache/
build-aux/
m4/
*.la
*.lo
*.o
*.tar.gz
Makefile
Makefile.in
aclocal.m4
config.h
config.h.in
config.log
config.status
configure
libtool
stamp-h1
# test suite
*.log
*.trs
# vmodtool
vcc_*_if.[ch]
vmod_*.rst
# man
*.1
*_options.rst
*_synopsis.rst
vmod_*.3
INSTALLATION
============
Building from source
~~~~~~~~~~~~~~~~~~~~
The VMOD is built on a system where an instance of Varnish is
installed, and the auto-tools will attempt to locate the Varnish
instance, and then pull in libraries and other support files from
there.
Quick start
-----------
This sequence should be enough in typical setups:
1. ``./bootstrap`` (for git-installation)
3. ``make``
4. ``make check`` (regression tests)
5. ``make install`` (may require root: sudo make install)
Alternative configs
-------------------
If you have installed Varnish to a non-standard directory, call
``autogen.sh`` and ``configure`` with ``PKG_CONFIG_PATH`` pointing to
the appropriate path. For example, when varnishd configure was called
with ``--prefix=$PREFIX``, use::
PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig
ACLOCAL_PATH=${PREFIX}/share/aclocal
export PKG_CONFIG_PATH ACLOCAL_PATH
Copyright 2023 UPLEX Nils Goroll Systemoptimierung
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the
distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
ACLOCAL_AMFLAGS = -I m4 -I @VARNISHAPI_DATAROOTDIR@/aclocal
DISTCHECK_CONFIGURE_FLAGS = RST2MAN=:
SUBDIRS = src
=========================================
A JSON formatter for VCL which sucks less
=========================================
.. role:: ref(emphasis)
.. _Varnish-Cache: https://varnish-cache.org/
This project provides JSON formatting facilities for `Varnish-Cache`_
VCL as a module (VMOD).
PROJECT RESOURCES
=================
* The primary repository is at https://code.uplex.de/uplex-varnish/libvmod-j
This server does not accept user registrations, so please use ...
* the mirror at https://gitlab.com/uplex/varnish/libvmod-j for issues,
merge requests and all other interactions.
INTRODUCTION
============
**BEFORE USING THIS VMOD, DO READ THIS DOCUMENTATION AND IN PARTICULAR
THE WARNING IN** `vmod_j(3) <src/vmod_j.man.rst>`_.
.. _JSON: https://www.json.org/json-en.html
Formatting `JSON`_ in pure VCL is a PITA, because string processing in
VCL was never made for it. VCL being a Domain Specific Language, it
was made for processing HTTP headers.
Consider this simple example of a JSON object with a single key
``key``, for which the value ``value`` is to be taken from a VCL
header variable::
{"key":"value"}
in pure VCL, you have to write something like this::
{"{"key":""} + req.http.value + {"""} + "}"
Applause if you do not lose your mental sanity trying to understand
what this does.
This vmod has been written because it drove the author crazy to
maintain VCL code with constructs like the above (and that's an
exceptionally trivial case).
DESCRIPTION (Excerpt)
=====================
(The full version is in `vmod_j(3) <src/vmod_j.man.rst>`_)
With VMOD *j*, the example above looks like this::
j.object("key" + req.http.value)
Things get more interesting with more complex data
structures. Consider this toy example::
j.object(
j.str("A") + j.null() +
"B" + j.object(
"BB" + j.number(string="42.42e42") +
"CC" + false +
"DD" + true) +
"C" + j.array("A" + 2 + j.object(j.nil()))
)
The JSON produced by this code looks like this when reformatted with
:ref:`jq(1)`::
{
"A": null,
"B": {
"BB": 4.242e+43,
"CC": false,
"DD": true
},
"C": [
"A",
2,
{}
]
}
Continue reading the documentation in `vmod_j(3) <src/vmod_j.man.rst>`_.
INSTALLATION
============
See `INSTALL.rst <INSTALL.rst>`_ in the source repository.
SUPPORT
=======
.. _gitlab.com issues: https://gitlab.com/uplex/varnish/libvmod-j/-/issues
To report bugs, use `gitlab.com issues`_.
For enquiries about professional service and support, please contact
info@uplex.de\ .
CONTRIBUTING
============
.. _merge requests on gitlab.com: https://gitlab.com/uplex/varnish/libvmod-j/-/merge_requests
To contribute to the project, please use `merge requests on gitlab.com`_.
To support the project's development and maintenance, there are
several options:
.. _paypal: https://www.paypal.com/donate/?hosted_button_id=BTA6YE2H5VSXA
.. _github sponsor: https://github.com/sponsors/nigoroll
* Donate money through `paypal`_. If you wish to receive a commercial
invoice, please add your details (address, email, any requirements
on the invoice text) to the message sent with your donation.
* Become a `github sponsor`_.
* Contact info@uplex.de to receive a commercial invoice for SWIFT payment.
COPYRIGHT
=========
::
Copyright 2023 UPLEX Nils Goroll Systemoptimierung
All rights reserved
Author: Nils Goroll <nils.goroll@uplex.de>
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions
are met:
1. Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
SUCH DAMAGE.
#!/bin/sh
set -e
set -u
WORK_DIR=$(pwd)
ROOT_DIR=$(dirname "$0")
cd "$ROOT_DIR"
if ! command -v libtoolize >/dev/null
then
echo "libtoolize: command not found, falling back to glibtoolize" >&2
alias libtoolize=glibtoolize
fi
mkdir -p m4
aclocal
libtoolize --copy --force
autoheader
automake --add-missing --copy --foreign
autoconf
cd "$WORK_DIR"
"$ROOT_DIR"/configure "$@"
AC_PREREQ([2.68])
AC_INIT([libvmod-vj], [0.1])
AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_AUX_DIR([build-aux])
AC_CONFIG_HEADERS([config.h])
AM_INIT_AUTOMAKE([1.12 -Wall -Werror foreign parallel-tests])
AM_SILENT_RULES([yes])
AC_ARG_WITH([rst2man],
AS_HELP_STRING(
[--with-rst2man=PATH],
[Location of rst2man (auto)]),
[RST2MAN="$withval"],
[AC_CHECK_PROGS(RST2MAN, [rst2man rst2man.py], [])])
VARNISH_PREREQ([6.0.0])
VARNISH_VMODS([j])
AM_PROG_AR
LT_PREREQ([2.2.6])
LT_INIT([dlopen disable-static])
AC_CONFIG_FILES([
Makefile
src/Makefile
])
AC_OUTPUT
AS_ECHO("
==== $PACKAGE_STRING ====
varnish: $VARNISH_VERSION
prefix: $prefix
vmoddir: $vmoddir
vcldir: $vcldir
pkgvcldir: $pkgvcldir
compiler: $CC
cflags: $CFLAGS
ldflags: $LDFLAGS
")
AM_CFLAGS = $(VARNISHAPI_CFLAGS)
# Modules
vmod_LTLIBRARIES = \
libvmod_j.la
libvmod_j_la_LDFLAGS = $(VMOD_LDFLAGS)
libvmod_j_la_SOURCES = vmod_j.c
nodist_libvmod_j_la_SOURCES = \
vcc_j_if.c \
vcc_j_if.h
@BUILD_VMOD_J@
# Test suite
AM_TESTS_ENVIRONMENT = \
PATH="$(abs_builddir):$(VARNISH_TEST_PATH):$(PATH)" \
LD_LIBRARY_PATH="$(VARNISH_LIBRARY_PATH)"
TEST_EXTENSIONS = .vtc
VTC_LOG_COMPILER = varnishtest -vl
AM_VTC_LOG_FLAGS = \
-p vcl_path="$(abs_top_srcdir)/vcl:$(VARNISHAPI_VCLDIR)" \
-p vmod_path="$(abs_builddir)/.libs:$(vmoddir):$(VARNISHAPI_VMODDIR)"
TESTS = \
vtc/vmod_j.vtc
# Documentation
dist_doc_DATA = \
vmod_j.vcc \
$(TESTS)
dist_man_MANS = \
vmod_j.3
.rst.1:
$(AM_V_GEN) $(RST2MAN) $< $@
.PHONY: flint
flint:
flexelint $(AM_CFLAGS) -I .. flint.lnt *.c
// must always be included to ensure sanity
-efile(766, config.h)
-e717 // do ... while(0)
// we can not change external interfaces / code
-efile(766, *_if.c)
// constructors not referenced
-esym(528, init_*)
-esym(528, assert_*)
/*-
* Copyright 2023 UPLEX Nils Goroll Systemoptimierung
* All rights reserved
*
* Author: Nils Goroll <nils.goroll@uplex.de>
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
* 1. Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright
* notice, this list of conditions and the following disclaimer in the
* documentation and/or other materials provided with the distribution.
*
* THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
* ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
* ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
* FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
* DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
* OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
* HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
* LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
* OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
* SUCH DAMAGE.
*/
#include "config.h"
#include <stdio.h>
#include <string.h>
#include <cache/cache.h>
#include <vsb.h>
#include "vcc_j_if.h"
static int
is_jnumber(const char *s)
{
if (*s == '-')
s++;
// 0 or [1-9][0-9]*
if (*s == '0')
s++;
else if (! (*s >= '1' && *s <= '9'))
return (0);
else {
// *s == 1..9
s++;
while (*s >= '0' && *s <= '9')
s++;
}
// fraction
if (*s == '.') {
s++;
while (*s >= '0' && *s <= '9')
s++;
}
// exponent
if (*s == 'e' || *s == 'E') {
s++;
if (*s == '-' || *s == '+')
s++;
while (*s >= '0' && *s <= '9')
s++;
}
if (*s != '\0')
return (0);
return (1);
}
static int
is_jlit(const char *s)
{
return (strcmp(s, "true") == 0 ||
strcmp(s, "false") == 0 ||
strcmp(s, "null") == 0);
}
// magic values
#define JM_object 0x81
#define JM_min JM_object
#define JM_array 0x82
#define JM_string 0x83
#define JM_number 0x84
#define JM_lit 0x85
#define JM_max JM_lit
#define AL __attribute__((aligned(2)))
static const unsigned char jm_object[2] AL = { JM_object, '{' };
static const unsigned char jm_array[2] AL = { JM_array, '[' };
static const unsigned char jm_string[2] AL = { JM_string, '"' };
static const unsigned char jnil_object[4] AL = { JM_object, '{','}', 0 };
static const unsigned char jnil_array[4] AL = { JM_array, '[',']', 0 };
static const unsigned char jnull[6] AL = { JM_lit, 'n','u','l','l', 0};
static const unsigned char jtrue[6] AL = { JM_lit, 't','r','u','e', 0};
static const unsigned char jfalse[7] AL = { JM_lit, 'f','a','l','s','e', 0};
#define VSB_mcat(vsb, x) AZ(VSB_bcat(vsb, x, sizeof(x)))
#define isJ(p, X) (((uintptr_t)(p) & 1) == 1 && \
memcmp((unsigned char *)(p) - 1, jm_ ## X, sizeof(jm_ ## X)) == 0)
#define is_Jobject(p) isJ(p, object)
#define is_Jarray(p) isJ(p, array)
#define is_Jstring(p) isJ(p, string)
// is this specific enough?
#define is_J(p) ((uintptr_t)(p) & 1 && \
(unsigned char)(p)[-1] >= JM_min && \
(unsigned char)(p)[-1] <= JM_max)
static void __attribute__((constructor))
assert_statics(void)
{
//lint --e{506}
assert(is_Jobject(jm_object + 1));
assert(is_Jobject(jnil_object + 1));
assert(is_Jarray(jm_array + 1));
assert(is_Jarray(jnil_array + 1));
assert(is_Jstring(jm_string + 1));
assert(is_J(jnull + 1));
assert(is_J(jtrue + 1));
assert(is_J(jfalse + 1));
}
static inline int
is_Jnumber(const char *p)
{
if (((uintptr_t)p & 1) == 0)
return (0);
if ((unsigned char)p[-1] != JM_number)
return (0);
if (*p == '-')
p++;
return (*p >= '0' && *p <= '9');
}
VCL_STRANDS
vmod_nil(VRT_CTX)
{
(void) ctx;
return (vrt_null_strands);
}
VCL_STRING
vmod_null(VRT_CTX)
{
(void) ctx;
return ((const char *)jnull + 1);
}
VCL_STRING
vmod_true(VRT_CTX)
{
(void) ctx;
return ((const char *)jtrue + 1);
}
VCL_STRING
vmod_false(VRT_CTX)
{
(void) ctx;
return ((const char *)jfalse + 1);
}
#define ints_sz (size_t)4
#define ints_lim (int)100
static char ints[ints_lim][ints_sz] AL;
static void __attribute__((constructor))
init_ints(void)
{
size_t u;
int r;
for (u = 0; u < ints_lim; u++) {
r = snprintf(ints[u], ints_sz, "%c%zu", JM_number, u);
assert((unsigned)r < ints_sz);
assert(is_Jnumber(ints[u] + 1));
}
}
VCL_STRING
vmod_number(VRT_CTX, struct VARGS(number)*a)
{
struct vsb vsb[1];
const char *p;
int valid_i, valid_d;
intmax_t i = 0;
double d = nan("");
CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
AN(a);
(void)a;
valid_i = a->valid_integer + a->valid_bytes;
valid_d = a->valid_real + a->valid_duration + a->valid_time;
if (valid_i + valid_d + a->valid_string != 1) {
VRT_fail(ctx, "j.number(): exactly one argument must be given");
return (NULL);
}
if (a->valid_string) {
if (! is_jnumber(a->string)) {
VRT_fail(ctx, "j.number(): string %s invalid" ,
a->string);
return (NULL);
}
}
else if (a->valid_integer)
i = a->integer;
else if (a->valid_bytes)
i = a->bytes;
else if (a->valid_real)
d = a->real;
else if (a->valid_duration)
d = a->duration;
else if (a->valid_time)
d = a->time;
else
WRONG("args invalid");
if (valid_i && ! VRT_INT_is_valid(i)) {
VRT_fail(ctx, "INT overflow converting to string (0x%jx)", i);
return (NULL);
}
else if (valid_d && ! VRT_REAL_is_valid(d)) {
VRT_fail(ctx, "REAL overflow converting to string (%e)", d);
return (NULL);
}
if (valid_i && i >= 0 && i < ints_lim)
return (ints[i] + 1);
// on %.15g: https://stackoverflow.com/questions/30658919/the-precision-of-printf-with-specifier-g/54162486#54162486
WS_VSB_new(vsb, ctx->ws);
AZ(VSB_putc(vsb, JM_number));
if (a->valid_string)
AZ(VSB_bcat(vsb, a->string, (ssize_t)strlen(a->string)));
else if (valid_i)
AZ(VSB_printf(vsb, "%jd", i));
else if (valid_d)
AZ(VSB_printf(vsb, "%.15g", d));
else
WRONG("valid_X");
p = WS_VSB_finish(vsb, ctx->ws, NULL);
if (p == NULL) {
VRT_fail(ctx, "j.number(): our of workspace");
return (NULL);
}
AZ((uintptr_t)p & 1);
return (p + 1);
}
// add json string
static void
vsbjstring(struct vsb *vsb, const char *p)
{
AZ(VSB_putc(vsb, '"'));
VSB_quote(vsb, p, -1, VSB_QUOTE_JSON);
AZ(VSB_putc(vsb, '"'));
}
VCL_STRING
vmod_string(VRT_CTX, VCL_STRANDS s)
{
struct vsb vsb[1];
const char *p;
int i;
CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
AN(s);
WS_VSB_new(vsb, ctx->ws);
VSB_mcat(vsb, jm_string); // "
for (i = 0; i < s->n; i++)
VSB_quote(vsb, s->p[i], -1, VSB_QUOTE_JSON);
AZ(VSB_putc(vsb, '"'));
p = WS_VSB_finish(vsb, ctx->ws, NULL);
if (p == NULL) {
VRT_fail(ctx, "j.string(): our of workspace");
return (NULL);
}
AZ((uintptr_t)p & 1);
return (p + 1);
}
// add a json value to the vsb:
// - json, number or literal as is
// - anything else as string
static void
vsbjvalue(struct vsb *vsb, const char *p)
{
if (! (is_J(p) || is_jnumber(p) || is_jlit(p)))
vsbjstring(vsb, p);
else
AZ(VSB_bcat(vsb, p, (ssize_t)strlen(p)));
}
VCL_STRING
vmod_array(VRT_CTX, VCL_STRANDS s)
{
struct vsb vsb[1];
const char *p;
int i;
CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
AN(s);
if (s->n == 0)
return ((const char *)jnil_array + 1);
for (i = 0; i < s->n; i++) {
if (s->p[i] != NULL)
continue;
VRT_fail(ctx, "Strand %d is NULL in j.array()", i);
return (NULL);
}
WS_VSB_new(vsb, ctx->ws);
VSB_mcat(vsb, jm_array); // [
vsbjvalue(vsb, s->p[0]);
for (i = 1; i < s->n; i++) {
AZ(VSB_putc(vsb, ','));
vsbjvalue(vsb, s->p[i]);
}
AZ(VSB_putc(vsb, ']'));
p = WS_VSB_finish(vsb, ctx->ws, NULL);
if (p == NULL) {
VRT_fail(ctx, "j.array(): our of workspace");
return (NULL);
}
AZ((uintptr_t)p & 1);
return (p + 1);
}
// add a json key:
// - json string as is
// - other json, number or literal fails
// - anything else as string
static int
vsbjkey(VRT_CTX, struct vsb *vsb, const char *p)
{
if (is_Jstring(p))
AZ(VSB_bcat(vsb, p, (ssize_t)strlen(p)));
else if (is_J(p) || is_jnumber(p) || is_jlit(p)) {
VRT_fail(ctx, "keys must be strings, got %s", p);
return (0);
}
else
vsbjstring(vsb, p);
return (1);
}
// add key: value
static int
vsbjkv(VRT_CTX, struct vsb *vsb, const char *k, const char *v)
{
if (! vsbjkey(ctx, vsb, k))
return (0);
AZ(VSB_putc(vsb, ':'));
vsbjvalue(vsb, v);
return (1);
}
VCL_STRING
vmod_object(VRT_CTX, VCL_STRANDS s)
{
struct vsb vsb[1];
const char *p;
int i;
CHECK_OBJ_NOTNULL(ctx, VRT_CTX_MAGIC);
AN(s);
if (s->n == 0)
return ((const char *)jnil_object + 1);
if (s->n & 1) {
VRT_fail(ctx, "j.object() needs an even number of strands");
return (NULL);
}
for (i = 0; i < s->n; i++) {
if (s->p[i] != NULL)
continue;
VRT_fail(ctx, "Strand %d is NULL in j.object()", i);
return (NULL);
}
assert(s->n >= 2);
WS_VSB_new(vsb, ctx->ws);
VSB_mcat(vsb, jm_object); // {
if (! vsbjkv(ctx, vsb, s->p[0], s->p[1])) {
WS_Release(ctx->ws, 0);
return (NULL);
}
for (i = 2; i < s->n; i += 2) {
assert((i & 1) == 0);
AZ(VSB_putc(vsb, ','));
//lint -e{679} trunc
if (vsbjkv(ctx, vsb, s->p[i], s->p[i + 1]))
continue;
WS_Release(ctx->ws, 0);
return (NULL);
}
AZ(VSB_putc(vsb, '}'));
p = WS_VSB_finish(vsb, ctx->ws, NULL);
if (p == NULL) {
VRT_fail(ctx, "j.object(): our of workspace");
return (NULL);
}
AZ((uintptr_t)p & 1);
return (p + 1);
}
#-
# Copyright 2023 UPLEX Nils Goroll Systemoptimierung
# All rights reserved
#
# Author: Nils Goroll <nils.goroll@uplex.de>
#
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions
# are met:
# 1. Redistributions of source code must retain the above copyright
# notice, this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
#
# THIS SOFTWARE IS PROVIDED BY THE AUTHOR AND CONTRIBUTORS ``AS IS'' AND
# ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL AUTHOR OR CONTRIBUTORS BE LIABLE
# FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
# DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
# OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
# HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
# LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
# OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
# SUCH DAMAGE.
$Module j 3 "A JSON formatter for VCL which sucks less"
$ABI vrt
INTRODUCTION
============
**BEFORE USING THIS VMOD, DO READ THIS DOCUMENTATION AND IN PARTICULAR
THE** `WARNING`_.
.. _JSON: https://www.json.org/json-en.html
Formatting `JSON`_ in pure VCL is a PITA, because string processing in
VCL was never made for it. VCL being a Domain Specific Language, it
was made for processing HTTP headers.
Consider this simple example of a JSON object with a single key
``key``, for which the value ``value`` is to be taken from a VCL
header variable::
{"key":"value"}
in pure VCL, you have to write something like this::
{"{"key":""} + req.http.value + {"""} + "}"
Applause if you do not lose your mental sanity trying to understand
what this does.
This vmod has been written because it drove the author crazy to
maintain VCL code with constructs like the above (and that's an
exceptionally trivial case).
DESCRIPTION
===========
This VMOD is called *A JSON formatter for VCL which sucks less* in the
hope to provide to VCL authors a tool which contributes to their
mental well being. As we will see, for the time being, it still sucks,
just *less*.
With VMOD *j*, the example above looks like this::
j.object("key" + req.http.value)
Notice how the concatenation operator ``+`` is "repurposed" for
separating arguments in lack of support for variable arguments. This
is one of the two cases of `creative` use of the Varnish VRT API, see
`IMPLEMENTATION NOTES`_ for more details.
Things get more interesting with more complex data
structures. Consider this toy example::
j.object(
j.str("A") + j.null() +
"B" + j.object(
"BB" + j.number(string="42.42e42") +
"CC" + false +
"DD" + true) +
"C" + j.array("A" + 2 + j.object(j.nil()))
)
The JSON produced by this code looks like this when reformatted with
:ref:`jq(1)`::
{
"A": null,
"B": {
"BB": 4.242e+43,
"CC": false,
"DD": true
},
"C": [
"A",
2,
{}
]
}
This example should contain almost all of the JSON syntax there is:
An outer object
* with a key ``A`` and no value,
* a key ``B`` whose value is another object
* with the keys ``AA``, ``BB`` and ``CC`` having as values a number
and the booleans ``false`` and ``true``,
* and a key ``C``, whose value is an array containing as elements a
string ``A``, the number 2 and an empty object.
.. _wrapper functions:
Simple JSON types
-----------------
The safe way to create the simple JSON types is to always use the
respective functions:
* `j.null()`_ to create a *null* value
* `j.true()`_ to create a *true* boolean
* `j.false()`_ to create a *false* boolean
* `j.number()`_ to create a number
* `j.string()`_ to create a string
As seen by VCL, all JSON types are of the STRING type, but to *j*
functions, they have additional type information, see `IMPLEMENTATION
NOTES`_ for details.
Auto typing
~~~~~~~~~~~
Alternative to the type functions above, native VCL expressions can be
used in many cases. As seen by VMOD *j*, all arguments to its
functions are strings as the result of *string folding* from the
original data types. VMOD *j* treats these strings as JSON types if
they look like them:
* ``"true"`` and ``"false"`` are considered boolean
* anything which looks like a number is considered a number
* anything else is considered a string.
So, for example, if you want to use a *JSON string* ``"true"``, use
``j.string(true)`` instead of ``"true"`` or ``true`` (which gets
folded into ``"true"`` by varnish).
Complex JSON types
------------------
`j.object()`_ creates objects. It always needs to be given an even
number of "arguments" (concatenated strings). Each even numbered
"argument" needs to be a JSON-string, so either be produced by
`j.string()`_ or treated like one by the rules given above.
Due to the VRT interface, the `j.nil()`_ argument needs to be used to
create an empty object: ``j.object(j.nil())`` creates ``{}``.
`j.array()`_ creates arrays. A `j.nil()`_ argument creates an empty
array. See above for pitfalls.
Unwrapped arguments to `j.object()`_ and `j.array()`_
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Arguments to `j.object()`_ and `j.array()`_ which are not passed
through the `wrapper functions`_ (literal strings, VCL variables or
other VMOD functions) are always interpreted as one of
* *boolean*,
* *number* or
* *string*,
but never as complex types.
For example ``j.object("key" + "[1,2,3]")`` creates ``{"key":
"[1,2,3]"}`` because the value is taken as a string.
To create ``{"key": [1,2,3]}``, use ``j.object("key" +
j.array(j.number(1) + j.number(2) + j.number(3)))``.
Pitfalls due to repurpose of the ``+`` operator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Our `creative` use of concatenated strings (STRANDS) as variable
arguments fails when varnish combines strings at compile time or when
the ``+`` operator is interpreted as arithmetic.
For example, ``j.array(1 + 2 + 3)`` produces ``[6]`` and
``j.array("1" + "2" + "3")`` produces ``[123]``.
This VMOD might suck less, but it still sucks, until, maybe, the VRT
interface gets more powerful.
As mentioned above, to create an array of three numbers (``[1,2,3]``),
use ``j.array(j.number(1) + j.number(2) + j.number(3))`` and to create
an array of three strings (``["1","2","3"]``) use
``j.array(j.string(1) + j.string(2) + j.string(3))``
VMOD INTERFACE REFERENCE
========================
$Function STRING null()
Returns a JSON *null* value.
$Function STRING true()
Returns a JSON *true* boolean.
$Function STRING false()
Returns a JSON *false* boolean.
$Function STRANDS nil()
Use this as an argument to create an empty `j.array()`_ or
`j.object()`_.
$Function STRING number([INT integer], [REAL real], [BYTES bytes],
[DURATION duration], [TIME time], [STRING string])
Return a JSON number created from exactly one of the optional
arguments. Calling this function with more than one argument triggers
a VCL failure.
* For the *integer* and *real* arguments, the returned number is the
same as the argument, formatted according to the JSON requirements,
* for *bytes*, the number of bytes,
* for *duration*, the number of seconds,
* for *time*, the number of seconds since 00:00 UTC on January 1st,
1970 ("UNIX epoch"),
* and for *string*, it is just that string, if it conforms to JSON
formatting.
$Alias num number
$Function STRING string(STRANDS)
Return the argument as a JSON string with quotes added and special
characters escaped appropriately.
$Alias str string
$Function STRING array(STRANDS)
Return a JSON array with the constituents of a concatenation argument
(strands) as elements.
Examples:
* to create ``[]``::
j.array(j.nil())
* to create ``[1]``::
j.array(j.number(1))
* to create ``["A", 42]``::
j.array(j.string("A") + j.number(42))
$Alias arr array
$Function STRING object(STRANDS)
Return a JSON object with pairs of the constituents of a
concatenation argument (strands) as key/value pairs.
* to create ``{}``::
j.array(j.nil())
* to create ``{"A":42}``::
j.object(j.string("A") + j.number(42))
* to create ``{"A":42,"B":true}``::
j.object(
j.string("A") + j.number(42) +
j.string("B") + j.true
)
$Alias obj object
WARNING
=======
This vmod makes use of implementation details of the current Varnish
VMOD interface, which might change any time. While the maintainer of
this VMOD will try to keep it working, *be prepared for breaking
changes*.
IMPLEMENTATION NOTES
====================
STRANDS as variable arguments
-----------------------------
This vmod uses the STRANDS data type of the Varnish VMOD interface as
a replacement for a variable arguments interface: When the VCC
compiler sees multiple, concatenated strings to make up a STRANDS
function argument, it tries to create a single string at compile time,
which is only possible if all strings are literal. Otherwise, it
creates a STRANDS with the values of the individual operands of the
concatenation.
This vmod makes use of this particular implementation: By wrapping the
operands of the concatenation with functions, the VCC compiler can not
create a literal string at compile time.
Magic values of JSON STRINGs
----------------------------
Having understood the above, consider this example::
j.object(j.string("A") + j.object(j.nil()))
produces ``{"A":{}}``, and::
j.object(j.string("A") + "{}")
produces ``{"A":"{}"}``.
If ``j.object(j.nil())`` produces the *VCL STRING* ``{}``, why does
``j.object(j.string("A") + j.object(j.nil()))`` not produce
``{"A":"{}"}``?
The answer is that the STRINGs returned by *j* functions always start
at an odd memory address and have a magic value in the preceding
byte. In memory, the return value of ``j.object(j.nil())`` looks like
this::
\x81{}
^
+--- pointer
This method relies on a couple of assumptions:
* Inspection of a pointer addresses is possible
* A pointer address can be decremented
* If an odd memory address is valid, that address minus one is also
valid.
These assumptions are true on modern hardware, but they might not be
forever.
For the magic check to do the right thing, we also rely on the fact
that ordinary strings produced by VCC or VMODs begin at an even
address (which most of them do) and, even if they begin at an odd
address, are highly likely to not be preceded by a value of 0x81 to
0x85.
In particular, all this is only relevant if literals are used. If the
`wrapper functions`_ are used always, the method is safe.
Only if un-wrapped literals are used, the worst that could happen is
that we would miss to auto-magically string-wrap an argument which
would need string wrapping to produce correct JSON or that we would
accept an invalid element as an object key.
SEE ALSO
========
vcl\(7),
varnishd\(1)
varnishtest "test vmod-j"
varnish v1 -vcl {
import std;
import j;
backend proforma none;
sub vcl_recv {
return (synth(200));
}
sub vcl_synth {
set resp.http.barf = {"{"key":""} + resp.reason + {"""} + "}";
set resp.http.easy = j.obj("key" + resp.reason);
# We need varargs...
set resp.http.still-sucks-1 = j.array(1 + 2 + 3);
set resp.http.still-sucks-2 = j.array("1" + "2" + "3");
set resp.http.unsuck-1 = j.array(j.number(1) + j.number(2) + j.number(3));
set resp.http.unsuck-2 = j.array(j.string(1) + j.string(2) + j.string(3));
#######
# null
set resp.http.null = j.null();
#######
# number
set resp.http.ni = j.number(integer=5);
set resp.http.nr = j.number(real=4.2);
set resp.http.nb = j.number(bytes=2kb);
set resp.http.nd = j.number(duration=1m);
set resp.http.nt = j.number(time=std.time(real=1693148786.742));
set resp.http.ns = j.number(string="-42.42e+42");
#######
# string
set resp.http.st = j.string(true);
#######
# array
set resp.http.a0 = j.array(j.nil());
set resp.http.a1 = j.array(j.number(1));
set resp.http.a2 = j.array(j.number(1) + j.null() +
j.number(-2222));
set resp.http.a3 = j.array(j.number(1) + "2.2e3" + 3);
set resp.http.a1a2 = j.array("A" + true + 1.2 +
j.array(j.number(1) + j.number(-2222)));
#######
# object
set resp.http.o0 = j.object(j.nil());
set resp.http.o1 = j.object("A" + 42);
set resp.http.o2 = j.object(
j.str("A") + j.null() +
"B" + "42.42e42" +
"C" + false +
"D" + true
);
set resp.http.o3 = j.object(
j.str("A") + j.null() +
"B" + j.object(
"BB" + j.number(string="42.42e42") +
"CC" + false +
"DD" + true) +
"C" + j.array("A" + 2 + j.object(j.nil()))
);
set resp.http.o4 = j.object(j.string("A") + j.object(j.nil()));
set resp.body = "";
return (deliver);
}
} -start
client c1 {
txreq
rxresp
expect resp.status == 200
expect resp.http.barf == "{\"key\":\"OK\"}"
expect resp.http.easy == resp.http.barf
expect resp.http.still-sucks-1 == "[6]"
expect resp.http.still-sucks-2 == "[123]"
expect resp.http.unsuck-1 == "[1,2,3]"
expect resp.http.unsuck-2 == "[\"1\",\"2\",\"3\"]"
expect resp.http.null == "null"
expect resp.http.ni == "5"
expect resp.http.nr == "4.2"
expect resp.http.nb == "2048"
expect resp.http.nd == "60"
expect resp.http.nt == "1693148786.742"
expect resp.http.ns == "-42.42e+42"
expect resp.http.st == "\"true\""
expect resp.http.a0 == "[]"
expect resp.http.a1 == "[1]"
expect resp.http.a2 == "[1,null,-2222]"
expect resp.http.a3 == "[1,2.2e3,3]"
expect resp.http.a1a2 == "[\"A\",true,1.200,[1,-2222]]"
expect resp.http.o0 == "{}"
expect resp.http.o1 == "{\"A\":42}"
expect resp.http.o2 == \
"{\"A\":null,\"B\":42.42e42,\"C\":false,\"D\":true}"
expect resp.http.o3 == \
"{\"A\":null,\"B\":{\"BB\":42.42e42,\"CC\":false,\"DD\":true},\"C\":[\"A\",2,{}]}"
expect resp.http.o4 == "{\"A\":{}}"
} -run
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment