In varnish-cache, the deliberate decision has been made to not support
TLS from the same address space as varnish itself, see
doc/sphinx/phk/ssl_again.rst

So the obvious way to connect to TLS backends is to use a TLS
"onloader" (a term coined by @slimhazard as in the opposite of
"offloader"), which turns a clear connection into a TLS connection.

Before this change, this required additional configuration in two
places: An address/port or UDS path needs to be uniquely allocated for
each destination address, the specific onloader configuration has to
be put in place and a varnish backend pointing to the onloader needs
to be added. All of this for each individual backend. Also, this
requirement prevents any use of dynamic backends with a TLS onloader.

haproxy, however, offers a convenient and elegant way to avoid this
configuration overhead: The PROXY protocol can also be used to
transport the destination address which haproxy is to connect to if a
server's address is unspecified (IN_ADDR_ANY / 0.0.0.0). The
configuration template for this use case looks like this (huge thank
you to @wtarreau for pointing out this great option in haproxy):

listen clear-to-ssl
        bind /my/path/to/ssl_onloader accept-proxy
        balance roundrobin
        stick-table type ip size 100
        stick on dst
        server s0 0.0.0.0:443 ssl ca-file /etc/ssl/certs/ca-bundle.crt
        server s1 0.0.0.0:443 ssl ca-file /etc/ssl/certs/ca-bundle.crt
        server s2 0.0.0.0:443 ssl ca-file /etc/ssl/certs/ca-bundle.crt
        # .. approximately as many servers as expected peers
        # for improved tls session caching

With this setup, by connecting to /my/path/to/ssl_onloader and sending
the address to make a TLS connection to in a PROXY header (as the
server address / port), we can reduce the configuration overhead
outside varnish substantially. In particular, we do not require a path
/ port per destination dynamic TLS backends become possible

This patch implements the basis for simple means of configuring such
an ssl onloader:

backends can be created with an additional "via" director, which has
to resolve to a simple backend. The connection is then made to that
address and the actual endpoint address is sent in an additional PROXY
header.  Notice that sending yet another proxy header to the actual
backend is unaffected. Despite using the same format, the two proxy
headers are semantically different: The first, here coined the
"preamble", is the address to make the connection to while the
(optional) second proxy header continues to contain the addresses of
the connection to varnish.

Future improvements on the roadmap:

* resolution of the "via" backend at the time the connection is made:
  This will allow for fault tolerance and load balancing of via
  backends

* Cascade the health check: If the "via" backend is probed / set down,
  any backends using it could be set unhealthy also.

* Timeouts: The "via" backend's timeouts could define maximum values
  for any connections made through it

Tivia:

To future Varnish-Cache historians, this patch originates from #2850
and went through three more iterations, making it a likely candidate
for the PR with the longest turnaround time of 1543 days.

The value should always be set, because we already assign zero on error,
but it might escape (pun intended) the vigilance of the compiler.

Instead of always making `VNUM_2bytes()` assign its second parameter, it
was decided that fixing the test case would be enough since we already
require error handling.

Document that the daemon and worker user have to share their primary
group.

Provide an exmaple of how to set up a system for the default users.

Note: I am well aware of the commands in pkg-varnish-cache, but they
use Linux specific useradd syntax. The commands given hopefully are
portable - I tested them on Solaris and Linux.

phk, if any of this does not match your intentions, please just change it back.

waving from the boat...

This reverts commit 6f50b7c8.

The test case was correct but too fast for the dispatch instance check.

If a dispatch server instance is already done by the time we list
servers with varnish -vcl+backend we end up with the condition failing
on the fd field being negative, since the session was already closed.

Adding an explicit flag will prevet that from happening.

This reverts commit 0c1aef58.

The close_range(2) system call is too recent and not recognized by the
host system on CircleCI, so the fedora-latest container detects it but
is denied execution (EPERM) from the host's libseccomp.

Also, on platforms with neither close_range(2) nor closefrom(2) we ended
up not including <dirent.h> and failing virtually everywhere in our CI.

The ifdef dance could have looked like this:

    #ifdef HAVE_LINUX_CLOSE_RANGE_H
    #  include <linux/close_range.h>
    #elif HAVE_CLOSEFROM
    #else
    #  include <dirent.h>
    #endif

Note the extra #else missing from the original patch.

This is reverted for now because we need to check that close_range(2)
works at configure time to circumvent the host mismatch problem.

I changed Walid's test to make it run faster and created an unfortunate
race condition as a result.

Considering that both Varnish and the backend should normally react to
the `Connection: close` header added by default, and considering how the
probe code is structured, the least intrusive approach is to tolerate a
timeout when we don't expect the backend to actively close the
connection.

Despite adding a `Connection: close` header by default to probe
requests, Varnish does not actively close the connection as it should.
This new attribute will allow to tolerate backends that equally don't
honor this header, and it is true by default to match the current
behavior.

which might come from libbsd piggy-bagged on libedit.

The current backend implementation reads the headers all at once, as a
big buffer, then manually chops them up, and later on, in the startfetch
step, Varnish loops through all the headers and prints them.

This is inconvenient for custom backends that are most likely going to
use http_SetH() (directly or via http_SetHeader(), http_PrinfHeader() or
others), which also prints the headers being added. As a result, those
implementations end up logging the header twice.

To work around the issue we can push the burden of logging the beresp
headers onto the backend implementation. It does change one test, as
now the Timestamp:Beresp log record appears after the headers instead
of before.

There's no point checking that a resp header is unset when we don't
rxresp in the first place... There were other things that could be
simplified.

Closes #3681
Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

In the unlikely event that we ever need actual coverage for this case,
we can handle it better than just failing the test case. In that regard,
as long as varnishtest itself can send SETTINGS frame on a wrong stream
to test varnishd's behavior, we should be fine.
Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

This will be useful as varnishtest will honor more settings.
Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

It groups the ws and iws fields together and hopefully conveys that
h2_window::init means "initial window size" slightly better than iws.
Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

This fields were never set in the first place so they went away in #3888.

We don't have SunOS coverage on Github so I noticed it after the facts.
I did look at the Solaris jail but somehow missed that those fields were
used there as well. Chances are that the deleted statements never ran in
the first place, otherwise the assertions would have triggered.

If the solaris jail should set[gu]id(2) as part of its privileges drop,
it should probably grow new sub-options similar to the ones in the unix
jail.

Refs #3888

This aligns with varnishd where durations are always computed in seconds
instead of introducing corner cases where sometimes it's milliseconds.

It also aligns with vtc_syslog that was introduced after the change to
"seconds everywhere" in varnishd.

We already pass it to VEV and VCLI subsystems in places where a
double is expected.

Trivia: we currently parse it in two distinct ways. So for now I'm
not eager to support duration units.

They are shared with the cache process but are never used. Only the VCC
process uses them, but they are never set. This specific fchown(2) call
in the VCC process was probably a no-op in the first place: since the
fields are never set this is transferring ownership to root:root and if
that succeeded the process was already root in the first place. If it
failed, we never see the error message since we lacked root privileges.

Both the unix and solaris jails are designed to run VCC (and CC) with
limited privileges, and in the absence of a jail, the outcome should be
the same: VCC creates a file with credentials suitable for the next CC
invocation.

As I suspected, the -Werror setup for libvgz C flags were needed to
properly discard them for suncc. Both warnings are conditionally not
turned into errors, so -Wno-unknown-warning-option should no longer
be needed.

Refs madler/zlib#633

Trying to fix the build for clang 15 actually broke the build for GCC.

The -Werror that was initially set after saving CFLAGS was meant to be
part of NO_VIZ test. We turn warnings into errors later in the configure
script so at this point we shouldn't care about it. If we really do, we
can move this check below the line where -Werror is set.

GCC chokes on -Wno-error=deprecated-non-prototype so instead we add it
conditionally.

To match the naming convention everywhere else, libvgz_extra_cflags was
renamed to VGZ_CFLAGS.

Refs 118fd10c

There are two warnings that we enforce for our own code that zlib does
not. There's also the visibility attribute that we check at configure
time. And regarding the visibility attribute, zlib no longer relies on
a NO_VIZ macro and aligned with the autoconf naming convention and wants
HAVE_HIDDEN instead.

Spotted by: Coverity