Major reorganization of the CircleCI workflows

In order to build aarch64 packages it relies on the machine executors
feature based on QEMU. Since the same scripts were needed to build
packages for different architectures they were extracted in standalone
shell scripts.

Alpine Linux packages are now built in addition to RPM and Deb packages
but considering how long emulation takes for aarch64 packages, packaging
was moved to a nightly workflow.

Some changes happened in the pkg-varnish-cache repository to accomodate
the CircleCI reorganization, mainly around dependency management.

More information in the CircleCI README file.

Closes #3313
Reviewed-by: Guillaume Quintard <guillaume@varnish-software.com>
Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

.circleci/README.rst

+Multiarch building, testing & packaging
+=======================================
+
+Varnish Cache uses CircleCI_ for building, testing and creating packages for
+several Linux distributions for both x86_64 and aarch64 architectures.
+
+Since CircleCI provides only x86_64 VMs the setup uses Docker and QEMU to be
+able to build, test and create packages for aarch64.  This is accomplished by
+registering ``qemu-user-static`` for the CircleCI ``machine`` executor::
+
+    sudo docker run --rm --privileged multiarch/qemu-user-static --reset --credential yes --persistent yes
+
+Note 1: **--credential yes** is needed so that *setuid* flag is working.
+Without it ``sudo`` does not work in the Docker containers with architecture
+different than x86_64.
+
+Note 2: **--persistent yes** is needed so that there is no need to use
+``:register`` tag. This way one can run locally pure foreign arch Docker
+images, like the official ``arm64v8/***`` ones.
+
+With QEMU registered each build step can start a Docker image for any of the
+supported architectures to execute the ``configure``, ``make``, package steps.
+
+Workflows
+---------
+
+There are two CircleCI workflows:
+
+commit
+~~~~~~
+
+It is executed after each push to any branch, including Pull Requests
+
+The ``commit`` workflow runs two jobs:
+
+- ``dist`` - this job creates the source code distribution of Varnish Cache as
+  compressed archive (``varnish-${VERSION}.tar.gz``).
+
+- ``distcheck`` - untars the source code distribution from ``dist`` job and
+  builds (*configure*, *make*) on different Linux distributions
+
+nightly
+~~~~~~~
+
+It is executed once per day at 04:00 AM UTC time.
+
+This workflow also builds binary packages for different Linux distributions
+and CPU architectures (x86_64 & aarch64) and for this reason its run takes
+longer.
+
+It runs the following jobs:
+
+- The first two jobs that run in parallel are:
+
+  - ``tar_pkg_tools`` - this step checks out pkg-varnish-cache_ with the
+    packaging descriptions for Debian, RedHat and Alpine, and stores them in
+    the build workspace for the next steps in the pipeline.
+
+  - ``dist`` - this step creates the source code distribution of Varnish Cache
+    as compressed archive (``varnish-${VERSION}.tar.gz``). This archive is
+    also stored in the build workspace and used later by the packaging steps.
+
+
+- The next job in the workflow is ``package`` - a job  that creates the
+  packages (e.g. .rpm, .deb) for each supported CPU architecture, Linux
+  distribution and its major version (e.g. *x64_centos_7*,
+  *aarch64_ubuntu_bionic*, *x64_alpine_3*, etc.). This step creates a
+  Dockerfile on the fly by using a base Docker image. This custom Docker image
+  executes a Shell script that has the recipe for creating the package for the
+  specific Linux flavor, e.g.  *make-rpm-packages.sh*. The step stores the
+  packages in the build workspace.
+
+- Finally, if the previous jobs are successful, a final step is executed -
+  ``collect_packages``. This step creates an archive with all packages and
+  stores it as an artifact that can be uploaded to PackageCloud_.
+
+
+More
+----
+
+This setup can be easily extended for any CPU architectures supported by QEMU
+and for any Linux distributions which have Docker image. To do this one needs
+to add a new ``package`` job with the proper parameters for it.
+
+At the moment the setup uses *raw* Docker images and installs the required
+Linux distribution dependencies before running the tests/build/packaging code.
+This could be optimized to save some execution time by creating custom Docker
+images that extend the current ones and pre-installs the required
+dependencies.
+
+.. _CircleCI: https://app.circleci.com/pipelines/github/varnishcache/varnish-cache
+.. _pkg-varnish-cache: https://github.com/varnishcache/pkg-varnish-cache
+.. _PackageCloud: https://packagecloud.io/varnishcache/

.circleci/make-apk-packages.sh

+#!/usr/bin/env sh
+
+set -eux
+
+apk add -q --no-progress --update tar alpine-sdk
+
+echo "PARAM_RELEASE: $PARAM_RELEASE"
+echo "PARAM_DIST: $PARAM_DIST"
+
+if [ -z "$PARAM_RELEASE" ]; then
+    echo "Env variable PARAM_RELEASE is not set! For example PARAM_RELEASE=8, for CentOS 8"
+    exit 1
+elif [ -z "$PARAM_DIST" ]; then
+    echo "Env variable PARAM_DIST is not set! For example PARAM_DIST=centos"
+    exit 1
+fi
+
+cd /varnish-cache
+tar xazf alpine.tar.gz --strip 1
+
+adduser -D builder
+echo "builder ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers
+addgroup builder abuild
+mkdir -p /var/cache/distfiles
+chmod -R a+w /var/cache/distfiles
+
+echo "Generate key"
+su builder -c "abuild-keygen -nai"
+
+echo "Fix APKBUILD's variables"
+tar xavf varnish-*.tar.gz
+VERSION=$(varnish-*/configure --version | awk 'NR == 1 {print $NF}')
+echo "Version: $VERSION"
+sed -i "s/@VERSION@/$VERSION/" APKBUILD
+rm -rf varnish-*/
+
+echo "Change the ownership so that abuild is able to write its logs"
+chown builder -R .
+echo "Fix checksums, build"
+su builder -c "abuild checksum"
+su builder -c "abuild -r"
+
+echo "Fix the APKBUILD's version"
+su builder -c "mkdir apks"
+ARCH=`uname -m`
+su builder -c "cp /home/builder/packages/$ARCH/*.apk apks"
+
+echo "Import the packages into the workspace"
+mkdir -p packages/$PARAM_DIST/$PARAM_RELEASE/$ARCH/
+mv /home/builder/packages/$ARCH/*.apk packages/$PARAM_DIST/$PARAM_RELEASE/$ARCH/
+
+echo "Allow to read the packages by 'circleci' user outside of Docker after 'chown builder -R .' above"
+chmod -R a+rwx .

.circleci/make-deb-packages.sh

+#!/usr/bin/env bash
+
+set -eux
+
+export DEBIAN_FRONTEND=noninteractive
+export DEBCONF_NONINTERACTIVE_SEEN=true
+apt-get update
+apt-get install -y dpkg-dev debhelper devscripts equivs pkg-config apt-utils
+
+echo "PARAM_RELEASE: $PARAM_RELEASE"
+echo "PARAM_DIST: $PARAM_DIST"
+
+
+if [ -z "$PARAM_RELEASE" ]; then
+    echo "Env variable PARAM_RELEASE is not set! For example PARAM_RELEASE=8, for CentOS 8"
+    exit 1
+elif [ -z "$PARAM_DIST" ]; then
+    echo "Env variable PARAM_DIST is not set! For example PARAM_DIST=centos"
+    exit 1
+fi
+
+cd /varnish-cache
+ls -la
+
+echo "Untar debian..."
+tar xavf debian.tar.gz
+
+echo "Untar orig..."
+tar xavf varnish-*.tar.gz --strip 1
+
+echo "Update changelog version..."
+if [ -e .is_weekly ]; then
+    WEEKLY='-weekly'
+else
+    WEEKLY=
+fi
+VERSION=$(./configure --version | awk 'NR == 1 {print $NF}')$WEEKLY~$PARAM_RELEASE
+sed -i -e "s|@VERSION@|$VERSION-1|"  "debian/changelog"
+
+echo "Install Build-Depends packages..."
+yes | mk-build-deps --install debian/control || true
+
+echo "Build the packages..."
+dpkg-buildpackage -us -uc -j16
+
+echo "Prepare the packages for storage..."
+mkdir -p packages/$PARAM_DIST/$PARAM_RELEASE/
+mv ../*.deb packages/$PARAM_DIST/$PARAM_RELEASE/

.circleci/make-rpm-packages.sh

+#!/usr/bin/env bash
+
+set -eux
+
+echo "PARAM_RELEASE: $PARAM_RELEASE"
+echo "PARAM_DIST: $PARAM_DIST"
+
+if [ -z "$PARAM_RELEASE" ]; then
+    echo "Env variable PARAM_RELEASE is not set! For example PARAM_RELEASE=8, for CentOS 8"
+    exit 1
+elif [ -z "$PARAM_DIST" ]; then
+    echo "Env variable PARAM_DIST is not set! For example PARAM_DIST=centos"
+    exit 1
+fi
+
+yum install -y epel-release
+
+if [ "$PARAM_DIST" = centos ]; then
+  if [ "$PARAM_RELEASE" = 8 ]; then
+      dnf install -y 'dnf-command(config-manager)'
+      yum config-manager --set-enabled PowerTools
+  fi
+fi
+
+yum install -y rpm-build yum-utils
+
+export DIST_DIR=build
+
+cd /varnish-cache
+rm -rf $DIST_DIR
+mkdir $DIST_DIR
+
+
+echo "Untar redhat..."
+tar xavf redhat.tar.gz -C $DIST_DIR
+
+echo "Untar orig..."
+tar xavf varnish-*.tar.gz -C $DIST_DIR --strip 1
+
+echo "Build Packages..."
+if [ -e .is_weekly ]; then
+    WEEKLY='.weekly'
+else
+    WEEKLY=
+fi
+VERSION=$("$DIST_DIR"/configure --version | awk 'NR == 1 {print $NF}')$WEEKLY
+
+cp -r -L "$DIST_DIR"/redhat/* "$DIST_DIR"/
+tar zcf "$DIST_DIR.tgz" --exclude "$DIST_DIR/redhat" "$DIST_DIR"/
+
+RPMVERSION="$VERSION"
+
+RESULT_DIR="rpms"
+CUR_DIR="$(pwd)"
+
+rpmbuild() {
+    command rpmbuild \
+        --define "_smp_mflags -j10" \
+        --define "_sourcedir $CUR_DIR" \
+        --define "_srcrpmdir $CUR_DIR/${RESULT_DIR}" \
+        --define "_rpmdir $CUR_DIR/${RESULT_DIR}" \
+        --define "versiontag ${RPMVERSION}" \
+        --define "releasetag 0.0" \
+        --define "srcname $DIST_DIR" \
+        --define "nocheck 1" \
+        "$@"
+}
+
+yum-builddep -y "$DIST_DIR"/redhat/varnish.spec
+rpmbuild -bs "$DIST_DIR"/redhat/varnish.spec
+rpmbuild --rebuild "$RESULT_DIR"/varnish-*.src.rpm
+
+echo "Prepare the packages for storage..."
+mkdir -p packages/$PARAM_DIST/$PARAM_RELEASE/
+mv rpms/*/*.rpm packages/$PARAM_DIST/$PARAM_RELEASE/