With onerror=abort, the request is aborted as with a bad return code.

With onerror=continue, the include remains empty

This already behaved like I expected it to, this vtc merely adds
an explicit test.

suggested by Dridi

Connect to s1 and s2 via v2.

Note on the v2 VCL: We use this varnish instance as a PROXY protocol
aware forwarder, which takes the address to connect to from the
incoming PROXY header (to mimic haproxy instead of requiring it).

Previously, we used debug.dyn(), but that does not work with two
different backends because it does not create different backend
instances, so connection pooling fails on this level, unrelated to the
actual test subject.

We avoid this issue by an explicit VCL implementation.

If the .via field is also set, then the value of .authority is set
as the authority TLV in the PROXY header. This gives the "true"
backend (usually the ssl-onloader) the opportunity to set the SNI
(HostName field) from the TLV value, for the TLS handshake with the
remote backend.

This mandates that PROXYv2 is always used with a via backend (since
only version 2 supports TLVs).

If the value of .authority is the empty string, then the TLV is not
sent. If .authority is not set for the backend, then fall back to
.host_header, which itself may have been a fallback to .host. Note
that if neither .authority nor .host_header is set, and .host is
set to an IP address, then the IP address is forwarded as the SNI
value, which is not permitted for HostName (RFC4366 ch 3.1). So
users are advised to set either .authority or .host_header, or set
.authority="", when .via is set.

Usage note with haproxy:

To enable sending SNI when haproxy is used as a TLS onloader, ``sni
fc_pp_authority`` needs to be used with the backend configuration.

Full usage example with haproxy 2.2:

listen sslon
	mode	tcp
	maxconn	1000
	bind	/shared/varnish_haproxy/haproxy_sslon accept-proxy mode 777
	stick-table type ip size 100
	stick	on dst
	server	s00 0.0.0.0:443 ssl ca-file /etc/ssl/certs/ca-bundle.crt alpn http/1.1 sni fc_pp_authority
	server	s01 0.0.0.0:443 ssl ca-file /etc/ssl/certs/ca-bundle.crt alpn http/1.1 sni fc_pp_authority
	# ...

A higher number of servers improves TLS session caching.

Due to the intialization order, only native vcl backends can be used
from vcl as via backends. As directors are defined in vcl_init which
gets to run only after the native backends have been initialized,
directors can not be refered to from a backend.

In varnish-cache, the deliberate decision has been made to not support
TLS from the same address space as varnish itself, see
doc/sphinx/phk/ssl_again.rst

So the obvious way to connect to TLS backends is to use a TLS
"onloader" (a term coined by @slimhazard as in the opposite of
"offloader"), which turns a clear connection into a TLS connection.

Before this change, this required additional configuration in two
places: An address/port or UDS path needs to be uniquely allocated for
each destination address, the specific onloader configuration has to
be put in place and a varnish backend pointing to the onloader needs
to be added. All of this for each individual backend. Also, this
requirement prevents any use of dynamic backends with a TLS onloader.

haproxy, however, offers a convenient and elegant way to avoid this
configuration overhead: The PROXY protocol can also be used to
transport the destination address which haproxy is to connect to if a
server's address is unspecified (IN_ADDR_ANY / 0.0.0.0). The
configuration template for this use case looks like this (huge thank
you to @wtarreau for pointing out this great option in haproxy):

listen clear-to-ssl
        bind /my/path/to/ssl_onloader accept-proxy
        balance roundrobin
        stick-table type ip size 100
        stick on dst
        server s0 0.0.0.0:443 ssl ca-file /etc/ssl/certs/ca-bundle.crt
        server s1 0.0.0.0:443 ssl ca-file /etc/ssl/certs/ca-bundle.crt
        server s2 0.0.0.0:443 ssl ca-file /etc/ssl/certs/ca-bundle.crt
        # .. approximately as many servers as expected peers
        # for improved tls session caching

With this setup, by connecting to /my/path/to/ssl_onloader and sending
the address to make a TLS connection to in a PROXY header (as the
server address / port), we can reduce the configuration overhead
outside varnish substantially. In particular, we do not require a path
/ port per destination dynamic TLS backends become possible

This patch implements the basis for simple means of configuring such
an ssl onloader:

backends can be created with an additional "via" director, which has
to resolve to a simple backend. The connection is then made to that
address and the actual endpoint address is sent in an additional PROXY
header.  Notice that sending yet another proxy header to the actual
backend is unaffected. Despite using the same format, the two proxy
headers are semantically different: The first, here coined the
"preamble", is the address to make the connection to while the
(optional) second proxy header continues to contain the addresses of
the connection to varnish.

Future improvements on the roadmap:

* resolution of the "via" backend at the time the connection is made:
  This will allow for fault tolerance and load balancing of via
  backends

* Cascade the health check: If the "via" backend is probed / set down,
  any backends using it could be set unhealthy also.

* Timeouts: The "via" backend's timeouts could define maximum values
  for any connections made through it

Tivia:

To future Varnish-Cache historians, this patch originates from #2850
and went through three more iterations, making it a likely candidate
for the PR with the longest turnaround time of 1543 days.

The value should always be set, because we already assign zero on error,
but it might escape (pun intended) the vigilance of the compiler.

Instead of always making `VNUM_2bytes()` assign its second parameter, it
was decided that fixing the test case would be enough since we already
require error handling.

Document that the daemon and worker user have to share their primary
group.

Provide an exmaple of how to set up a system for the default users.

Note: I am well aware of the commands in pkg-varnish-cache, but they
use Linux specific useradd syntax. The commands given hopefully are
portable - I tested them on Solaris and Linux.

phk, if any of this does not match your intentions, please just change it back.

waving from the boat...

This reverts commit 6f50b7c8.

The test case was correct but too fast for the dispatch instance check.

If a dispatch server instance is already done by the time we list
servers with varnish -vcl+backend we end up with the condition failing
on the fd field being negative, since the session was already closed.

Adding an explicit flag will prevet that from happening.

This reverts commit 0c1aef58.

The close_range(2) system call is too recent and not recognized by the
host system on CircleCI, so the fedora-latest container detects it but
is denied execution (EPERM) from the host's libseccomp.

Also, on platforms with neither close_range(2) nor closefrom(2) we ended
up not including <dirent.h> and failing virtually everywhere in our CI.

The ifdef dance could have looked like this:

    #ifdef HAVE_LINUX_CLOSE_RANGE_H
    #  include <linux/close_range.h>
    #elif HAVE_CLOSEFROM
    #else
    #  include <dirent.h>
    #endif

Note the extra #else missing from the original patch.

This is reverted for now because we need to check that close_range(2)
works at configure time to circumvent the host mismatch problem.

I changed Walid's test to make it run faster and created an unfortunate
race condition as a result.

Considering that both Varnish and the backend should normally react to
the `Connection: close` header added by default, and considering how the
probe code is structured, the least intrusive approach is to tolerate a
timeout when we don't expect the backend to actively close the
connection.

Despite adding a `Connection: close` header by default to probe
requests, Varnish does not actively close the connection as it should.
This new attribute will allow to tolerate backends that equally don't
honor this header, and it is true by default to match the current
behavior.

which might come from libbsd piggy-bagged on libedit.

The current backend implementation reads the headers all at once, as a
big buffer, then manually chops them up, and later on, in the startfetch
step, Varnish loops through all the headers and prints them.

This is inconvenient for custom backends that are most likely going to
use http_SetH() (directly or via http_SetHeader(), http_PrinfHeader() or
others), which also prints the headers being added. As a result, those
implementations end up logging the header twice.

To work around the issue we can push the burden of logging the beresp
headers onto the backend implementation. It does change one test, as
now the Timestamp:Beresp log record appears after the headers instead
of before.

There's no point checking that a resp header is unset when we don't
rxresp in the first place... There were other things that could be
simplified.

Closes #3681
Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

In the unlikely event that we ever need actual coverage for this case,
we can handle it better than just failing the test case. In that regard,
as long as varnishtest itself can send SETTINGS frame on a wrong stream
to test varnishd's behavior, we should be fine.
Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

This will be useful as varnishtest will honor more settings.
Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

It groups the ws and iws fields together and hopefully conveys that
h2_window::init means "initial window size" slightly better than iws.
Signed-off-by: Dridi Boukelmoune <dridi.boukelmoune@gmail.com>

This fields were never set in the first place so they went away in #3888.

We don't have SunOS coverage on Github so I noticed it after the facts.
I did look at the Solaris jail but somehow missed that those fields were
used there as well. Chances are that the deleted statements never ran in
the first place, otherwise the assertions would have triggered.

If the solaris jail should set[gu]id(2) as part of its privileges drop,
it should probably grow new sub-options similar to the ones in the unix
jail.

Refs #3888