Unverified Commit 58ec40f9 authored by Nils Goroll's avatar Nils Goroll

Initial public release

parents
# editors
*~
# general
core.*
# build system
.deps/
.libs/
autom4te.cache/
build-aux/
m4/
*.la
*.lo
*.o
*.tar.gz
Makefile
Makefile.in
aclocal.m4
config.h
config.h.in
config.log
config.status
configure
libtool
stamp-h1
# test suite
*.log
*.trs
fellowfile
# vmodtool
vcc_*_if.[ch]
vmod_slash.rst
# vsctool
VSC_*.[ch]
VSC_*.rst
# dot
src/fcs_states.dot
# man
*.1
*.7
*_options.rst
*_synopsis.rst
vmod_*.3
slashmap.rst
# bin
slashmap
slash
buddy_test
buddy_test_witness
buddy_test_when
fellow_log_test
fellow_log_test_ndebug
fellow_cache_test
fellow_log_dbg
# coverage
*.gcda
*.gcno
/src/coverage
The slash project is provided under:
SPDX-License-Identifier: LGPL-2.1-only
with the exception of some files taken from the Varnish-Cache project
which are licensed under
SPDX-License-Identifier: BSD-2-Clause
Both licences are provided in the LICENSES directory.
\ No newline at end of file
================================
SLASH/ Installation Instructions
================================
.. role:: ref(emphasis)
Installing from Source
======================
.. _Compiling Varnish from source: https://varnish-cache.org/docs/trunk/installation/install_source.html
generic
-------
* Compile and install varnish-cache 7.2.0 or later by the
`Compiling Varnish from source`_ documentation.
* Optionally, install the following additional prerequisites:
recommendations for optimal fellow storage performance
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
.. _liburing: https://github.com/axboe/liburing.git
.. _xxhash: https://github.com/Cyan4973/xxHash
* On Linux, it is strongly recommended to add io_uring support by
installing kernel >= 5.10 (better: >= 5.13) and `liburing`_.
* For optimal performance, it is recommended to install `xxhash`_ and
later use ``xxh3_64`` or ``xxh3_128`` by setting the ``hash_obj``
and ``hash_log`` tuneables.
Note that a fellow storage using any of the `xxhash`_ hashes can
only be loaded by an instance with `xxhash`_ support compiled in.
compiling
~~~~~~~~~
* set and export the ``VARNISHSRC`` environment variable to point to
the varnish-cache source tree
* In the ``slash`` source tree, run::
./bootstrap
make check
make install
The last step will probably need to be run as root (e.g. using ``sudo``).
When installation has succeeded, continue with the `Next Steps`_.
Debian/Ubuntu
-------------
This guide was written based on:
* Debian 11 base install
* Varnish-Cache binary packages
Add the packagecloud repository, if not already done::
sudo apt-get install curl
curl -sO \
https://packagecloud.io/install/repositories/varnishcache/varnish72/script.deb.sh
# REVIEW script.deb.sh and continue only if considered safe
sudo bash script.deb.sh
Install varnish-cache dev package and other required packages::
sudo apt-get install -y \
varnish-dev \
git \
autoconf \
autoconf-archive \
automake \
libtool \
gcc \
python3-docutils \
python3-sphinx \
libpcre2-dev \
libreadline-dev \
liburing-dev \
libxxhash-dev \
libncurses-dev
Download and extract the varnish-cache source code for the installed
version, and export the ``VARNISHSRC`` environment variable to the right
directory::
v=$(/usr/sbin/varnishd -V 2>&1 | grep -Eo 'varnish-[0-9.]+')
curl -LO \
https://github.com/varnishcache/varnish-cache/archive/refs/tags/${v}.tar.gz
tar xf ${v}.tar.gz
VARNISHSRC=$(realpath varnish-cache-${v})
export VARNISHSRC
Build required components from varnish-cache::
( pushd $VARNISHSRC && \
./autogen.des && \
pushd include && \
make vcs_version.h && \
popd && pushd lib/libvarnish && \
make libvarnish.la && \
popd && pushd lib/libvsc &&
make VSC_main.c \
)
Clone this repository::
git clone http://code.uplex.de/uplex-varnish/slash.git
Enter the ``slash`` directory and call ``bootstrap``::
cd slash
./bootstrap
A summary similar to the one shown below should be output. In
particular, ``xxhash: yes`` and ``uring: yes`` should be contained::
==== slash 0.9 ====
varnish: 7.2.1
prefix: /usr
vmoddir: /usr/lib/varnish/vmods
vcldir: ${datarootdir}/varnish/vcl
pkgvcldir: ${vcldir}/${PACKAGE}
compiler: gcc
cflags: -g -O2 -fstack-protector-all
ldflags:
xxhash: yes
uring: yes
aio: no
Build, test and install SLASH/::
make check && sudo make install
Configuration options
---------------------
.. _autoconf: https://www.gnu.org/software/autoconf/
Among other things, ``bootstrap`` invokes the ``configure`` script
generated by `autoconf`_, and any command-line options given with
``bootstrap`` are passed on to ``configure``. For example,
``VARNISHSRC`` can be specified like this (since ``bootstrap`` is the
only step that requires it)::
bootstrap VARNISHSRC=/path/to/varnish-cache
CLI options can also be used, for example, to set ``CFLAGS``, select
the C compiler (with ``CC``), or specify the installation prefix.
``bootstrap --help`` (which invokes ``configure --help``) shows all of
the relevant variables and available options.
For most installations, the default values (with no CLI options, as
shown above) are sufficient.
Next Steps
----------
.. _Next Steps:
Now you should be ready to go.
Consult the :ref:`vmod_slash(3)` and :ref:`slashmap(1)` man pages for
detailed information on how to configure, use and monitor the SLASH/
storage engines.
System Setup for Optimal Performance
====================================
buddy & fellow memory cache
---------------------------
For the time being, these recommendations only apply to Linux.
The `buddy` allocator, which is also used by the memory cache of
`fellow`, will, by default, attempt to allocate memory from huge pages
and try to lock pages in RAM and thus prevent paging.
huge pages
~~~~~~~~~~
To the best of our knowledge, best performance is thus achieved by
ensuring that the system has sufficient huge pages reserved by
* either using the kernel command line, for example by adding to
``/etc/default/grub``::
GRUB_CMDLINE_LINUX="hugepagesz=1GB hugepages=300"
to configure 300GB of 1GB huge pages at boot and rebooting
* or by applying the same configuration at runtime through::
echo 300 \
>/sys/kernel/mm/hugepages/hugepages-1048576kB/nr_hugepages
to, again, configure 300GB of 1GB huge pages.
The 300GB used in these examples have to be adjusted approriately to
the desired memory cache size.
Alternatively, 2MB huge pages can be used where 1GB pages are not
available.
locked memory
~~~~~~~~~~~~~
:ref:`varnishd(1)` should be started with a limit on locked memory
higher than the configured memory size. We recommend to configure no
limit on locked memory by calling::
ulimit -l unlimited
or using::
LimitMEMLOCK=infinity
with :ref:`systemd(1)`, or whatever appropriate method to lift the
limit on locked memory.
fellow storage
--------------
`fellow` uses its own memory caching strategy, so double caching by
the operating system should be avoided. `fellow` thus tries to open
the storage path with ``O_DIRECT``, where available.
Nonetheless, to the best of our knowledge, avoiding the file system
overhead results in optimal performance. Where possible, the `fellow`
*path* should therefore point to a disk storage device. Using a mirror
or mirror/stripe (RAID1 or RAID10) is recommended for optimal
resilience.
As the IO pattern of `fellow` usually is highly random and because
access times are crucial for overall latency, using storage with the
lowest possible access times for random access is highly
recommended. Flash based storage (SSD, NVMe) usually falls into this
category.
Valid-License-Identifier: BSD-2-Clause
SPDX-URL: https://spdx.org/licenses/BSD-2-Clause.html
Usage-Guide:
To use the BSD 2-clause "Simplified" License put the following SPDX
tag/value pair into a comment according to the placement guidelines in
the licensing rules documentation:
SPDX-License-Identifier: BSD-2-Clause
License-Text:
Copyright (c) <year> <owner> . All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are met:
1. Redistributions of source code must retain the above copyright notice,
this list of conditions and the following disclaimer.
2. Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE
LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
POSSIBILITY OF SUCH DAMAGE.
This diff is collapsed.
ACLOCAL_AMFLAGS = -I m4 -I @VARNISHAPI_DATAROOTDIR@/aclocal
DISTCHECK_CONFIGURE_FLAGS = RST2MAN=:
SUBDIRS = src
dist_doc_DATA = README.rst COPYING LICENSES/BSD-2-Clause LICENSES/LGPL-2.1
EXTRA_DIST = INSTALL.rst
coverage:
$(MAKE) $(AM_MAKEFLAGS) -C src coverage
This diff is collapsed.
#!/bin/sh
set -e
set -u
WORK_DIR=$(pwd)
ROOT_DIR=$(dirname "$0")
cd "$ROOT_DIR"
if ! command -v libtoolize >/dev/null
then
echo "libtoolize: command not found, falling back to glibtoolize" >&2
alias libtoolize=glibtoolize
fi
mkdir -p m4
aclocal
libtoolize --copy --force
autoheader
automake --add-missing --copy --foreign
autoconf
cd "$WORK_DIR"
"$ROOT_DIR"/configure "$@"
AC_PREREQ([2.68])
AC_INIT([slash],[0.9])
AC_CONFIG_MACRO_DIR([m4])
AC_CONFIG_AUX_DIR([build-aux])
AC_CONFIG_HEADERS([config.h])
AC_USE_SYSTEM_EXTENSIONS
AM_INIT_AUTOMAKE([1.12 -Wall -Werror foreign parallel-tests])
AM_SILENT_RULES([yes])
AM_PROG_AR
AM_PROG_CC_C_O
AX_PTHREAD(,[AC_MSG_ERROR([Could not configure pthreads support])])
LIBS="$PTHREAD_LIBS $LIBS"
CFLAGS="$CFLAGS $PTHREAD_CFLAGS"
CC="$PTHREAD_CC"
LT_PREREQ([2.2.6])
LT_INIT([dlopen disable-static])
AC_ARG_WITH([rst2man],
AS_HELP_STRING(
[--with-rst2man=PATH],
[Location of rst2man (auto)]),
[RST2MAN="$withval"],
[AC_CHECK_PROGS(RST2MAN, [rst2man rst2man.py], [])])
AC_ARG_WITH([lcov],
AS_HELP_STRING(
[--with-lcov=PATH],
[Location of lcov to generate coverage data (auto)]),
[LCOV="$withval"],
[AC_CHECK_PROGS(LCOV, [lcov], [])])
AM_CONDITIONAL(HAVE_LCOV, [test -n "$LCOV"])
AC_ARG_WITH([genhtml],
AS_HELP_STRING(
[--with-genhtml=PATH],
[Location of genhtml to generate coverage reports (auto)]),
[GENHTML="$withval"],
[AC_CHECK_PROGS(GENHTML, [genhtml], [])])
AM_CONDITIONAL(HAVE_GENHTML, [test -n "$GENHTML"])
VARNISH_PREREQ([7.2.0])
VARNISH_VMODS([slash])
VARNISH_COUNTERS([slash])
VARNISH_UTILITIES([slashmap])
AC_ARG_VAR([VARNISHSRC], [path to Varnish source])
if test "x$VARNISHSRC" = x; then
AC_MSG_FAILURE([Need VARNISHSRC])
fi
VARNISHSRC_CFLAGS="$VARNISHAPI_CFLAGS \
-I$VARNISHSRC/bin/varnishd \
-I$VARNISHSRC/lib/libvsc"
AC_SUBST([VARNISHSRC_CFLAGS])
AC_CHECK_HEADERS([stdatomic.h atomic.h])
AC_CHECK_HEADERS([linux/mman.h linux/fs.h])
AC_CHECK_HEADERS([sys/dkio.h])
AC_ARG_ENABLE([uring],
[AS_HELP_STRING([--disable-uring],
[disable uring support])],
[],
[enable_uring=yes])
AS_IF([test "x$enable_uring" != xno],
[AC_CHECK_HEADERS([liburing.h])],
[ac_cv_header_liburing_h=no])
AC_ARG_ENABLE([asynch],
[AS_HELP_STRING([--disable-asynch],
[disable asynch support])],
[],
[enable_asynch=yes])
AS_IF([test "x$enable_asynch" != xno],
[AC_CHECK_HEADERS([sys/asynch.h])],
[ac_cv_header_sys_asynch_h=no])
AM_CONDITIONAL(HAVE_URING,[test "x$ac_cv_header_liburing_h" != "xno"])
AM_CONDITIONAL(HAVE_ASYNCH,[test "x$ac_cv_header_sys_asynch_h" != "xno"])
AC_CHECK_FUNCS([fallocate])
PKG_CHECK_MODULES([XXHASH], [libxxhash], [], [true])
# avoiding the fruitless RPATH debate for now
if ! echo "$pkg_cv_XXHASH_LIBS" | grep -- "-R" ; then
tmp=$(echo "$pkg_cv_XXHASH_LIBS" | grep -Po '(?<=-L)\S+')
if test -n "${tmp}" ; then
XXHASH_LIBS="${pkg_cv_XXHASH_LIBS} -R${tmp}"
fi
fi
AC_CHECK_HEADERS([xxhash.h])
## BEGIN from varnish-cache
# --enable-stack-protector XXX remove -all
AC_ARG_ENABLE(stack-protector,
AS_HELP_STRING([--enable-stack-protector],[enable stack protector (default is YES)]),
[],
[enable_stack_protector=yes])
if test "x$enable_stack_protector" != "xno"; then
AX_CHECK_COMPILE_FLAG([-fstack-protector-all],
AX_CHECK_LINK_FLAG([-fstack-protector-all],
[CFLAGS="${CFLAGS} -fstack-protector-all"], [], []),
[], [])
fi
PKG_CHECK_MODULES([CURSES], [ncursesw], [], [
PKG_CHECK_MODULES([CURSES], [ncurses], [], [
PKG_CHECK_MODULES([CURSES], [curses], [], [
AX_WITH_CURSES
if test "x$ax_cv_curses" != xyes; then
AC_MSG_ERROR([requires an X/Open-compatible Curses library])
fi
CURSES_LIBS="$CURSES_LIB"
])
])
])
AC_SUBST([CURSES_CFLAGS])
AC_SUBST([CURSES_LIBS])
save_CFLAGS="${CFLAGS}"
CFLAGS="${CFLAGS} ${CURSES_CFLAGS}"
AC_CHECK_HEADERS([ncursesw/curses.h ncursesw.h ncurses/curses.h ncurses.h curses.h])
CFLAGS="${save_CFLAGS}"
save_CFLAGS="${CFLAGS}"
CFLAGS="${CFLAGS} ${VARNISHAPI_CFLAGS}"
# varnish-cache compat with 7.2
AC_CHECK_MEMBER([struct boc.fetched_so_far],
[AC_DEFINE([BOC_FETCHED_SO_FAR], [fetched_so_far], [Name of the field])],
[AC_DEFINE([BOC_FETCHED_SO_FAR], [len_so_far], [Name of the field])],
[[#include <cache/cache.h>]])
CFLAGS="${save_CFLAGS}"
## END
AC_CONFIG_FILES([
Makefile
src/Makefile
src/slashmap.rst
])
AC_OUTPUT
AS_ECHO("
==== $PACKAGE_STRING ====
varnish: $VARNISH_VERSION
prefix: $prefix
vmoddir: $vmoddir
vcldir: $vcldir
pkgvcldir: $pkgvcldir
compiler: $CC
cflags: $CFLAGS
ldflags: $LDFLAGS
xxhash: $ac_cv_header_xxhash_h
uring: $ac_cv_header_liburing_h
aio: $ac_cv_header_sys_asynch_h
")
AM_CFLAGS = $(VARNISHAPI_CFLAGS) -fno-strict-aliasing
FREEPAGE_CFLAGS = -DFREEPAGE_WHEN
DEV_CFLAGS = -Wall -Werror
## to compile all with witness
#DEV_CFLAGS = -Wall -Werror -DBUDDY_WITNESS -DBUDDY_WITNESS_BACKTRACE
# Modules
vmod_LTLIBRARIES = \
libvmod_slash.la
# buddy libs
buddy_sources = \
buddy_witness.h \
slashmap.h \
buddy.h \
buddy_util.h \
buddy.c \
bitsof.h \
bitf.h \
tbl/mmap_flags.h \
foreign/vmb.h
noinst_LTLIBRARIES = \
libbuddy.la \
libbuddywhen.la \
libbuddywitness.la
libbuddy_la_CFLAGS = $(AM_CFLAGS) $(DEV_CFLAGS)
libbuddywhen_la_CFLAGS = $(libbuddy_la_CFLAGS) $(FREEPAGE_CFLAGS)
libbuddywitness_la_CFLAGS = $(libbuddy_la_CFLAGS) -DBUDDY_WITNESS
libbuddy_la_SOURCES = $(buddy_sources)
libbuddywhen_la_SOURCES = $(libbuddy_la_SOURCES)
libbuddywitness_la_SOURCES = $(libbuddy_la_SOURCES)
# vmod
# freepage is only used for buddy_storage, not for fellow_storage
noinst_LTLIBRARIES += \
libbuddy_storage.la
libbuddy_storage_la_CFLAGS = \
$(AM_CFLAGS) $(FREEPAGE_CFLAGS) $(VARNISHSRC_CFLAGS) $(DEV_CFLAGS)
libbuddy_storage_la_SOURCES = \
buddy_storage.c \
tbl/buddy_tunables.h \
buddy_tune.h \
buddy_tune.c
libvmod_slash_la_CFLAGS = $(AM_CFLAGS) $(VARNISHSRC_CFLAGS) $(DEV_CFLAGS)
libvmod_slash_la_LDFLAGS = $(VMOD_LDFLAGS)
libvmod_slash_la_SOURCES = \
vmod_slash.c \
vmod_slash_loadmasters.c \
fellow_storage.c \
fellow_storage_deref.h \
fellow_cache_storage.h \
fellow_stash.h \
fellow_log_storage.h
libvmod_slash_la_LIBADD = \
libbuddy_storage.la \
libbuddywhen.la \
libfellow.la
# XXX revisit
# linked to libfellow.la:
# libbuddy.la
nodist_libvmod_slash_la_SOURCES = \
vcc_slash_if.c \
vcc_slash_if.h
vmod_slash.c: vcc_slash_if.h
@BUILD_VMOD_SLASH@
# Utilities
bin_PROGRAMS = \
slashmap \
buddy_test \
buddy_test_when
slashmap_LDFLAGS = $(VARNISHAPI_LIBS) @CURSES_LIBS@
slashmap_CFLAGS = $(AM_CFLAGS) @CURSES_CFLAGS@ $(DEV_CFLAGS)
slashmap_SOURCES = \
slashmap.c \
slashmap_options.h \
slashmap.h \
pow2_units.c \
pow2_units.h \
bitsof.h \
bitf.h \
foreign/vcurses.h
buddy_test_LDFLAGS = $(VARNISHAPI_LIBS)
buddy_test_CFLAGS = \
$(AM_CFLAGS) -DDEBUG -DTEST_DRIVER $(DEV_CFLAGS)
buddy_test_SOURCES = \
$(buddy_sources)
buddy_test_witness_LDFLAGS = $(VARNISHAPI_LIBS)
buddy_test_witness_CFLAGS = \
$(AM_CFLAGS) -DDEBUG -DTEST_DRIVER -DBUDDY_WITNESS $(DEV_CFLAGS)
buddy_test_witness_SOURCES = \
$(buddy_sources)
buddy_test_when_LDFLAGS = $(VARNISHAPI_LIBS)
buddy_test_when_CFLAGS = \
$(AM_CFLAGS) -DDEBUG -DTEST_DRIVER $(FREEPAGE_CFLAGS) $(DEV_CFLAGS)
buddy_test_when_SOURCES = \
$(buddy_sources)
noinst_PROGRAMS = \
buddy_test \
buddy_test_witness \
buddy_test_when
# Test suite
AM_TESTS_ENVIRONMENT = \
PATH="$(abs_builddir):$(VARNISH_TEST_PATH):$(PATH)" \
LD_LIBRARY_PATH="$(VARNISH_LIBRARY_PATH)"
TEST_EXTENSIONS = .vtc
VTC_LOG_COMPILER = varnishtest -l -v -b 10m
AM_VTC_LOG_FLAGS = \
-D libvmod_slash="$(abs_builddir)/.libs/libvmod_slash.so" \
-p vcl_path="$(abs_top_srcdir)/vcl:$(VARNISHAPI_VCLDIR)" \
-p vmod_path="$(abs_builddir)/.libs:$(vmoddir):$(VARNISHAPI_VMODDIR)"
TESTS = \
buddy_test \
buddy_test_witness \
buddy_test_when \
vtc/buddy_vcl.vtc \
vtc/buddy_global.vtc \
vtc/buddy_edgecases.vtc \
vtc/buddy_c00062.vtc \
vtc/buddy_c00075.vtc \
vtc/buddy_c00097.vtc \
vtc/buddy_t02005.vtc \
vtc/fellow_vcl.vtc \
vtc/fellow_global.vtc \
vtc/fellow_global_shared.vtc \
vtc/fellow_coverage.vtc \
vtc/fellow_t02005.vtc \
vtc/vut_slashmap.vtc \
vtc/loadmasters.vtc
fellow_state_tables = \
tbl/fco_states.h \
tbl/fcs_states.h \
tbl/fco_transition.h \
tbl/fcs_transition.h
fellow_sources = \
pow2_units.c \
pow2_units.h \
buddy.h \
buddy_util.h \
compiler.h \
$(fellow_state_tables) \
fellow_debug.h \
fellow_io_backend.h \