Fixes: #1772

With VBT_Close now being capable of dealing with STOLEN connections, we
no longer need to VBT_Wait for them prior to close.

The change from shutdown(.., SHUT_WR) to shutdown(.., SHUT_RDWR) is
required to make it trigger a waiter event.

This test case should not rely on
first_byte_timeout/between_bytes_timeout.

The second time around, we force a fresh connection.

The VCL user may choose to do 'return (retry);' in vcl_backend_error{}
if further attempts are deemed warranted.

Fixes: #2135

Passing a nonzero value will force a fresh backend connection.

We only want to return the connection early to the waiter when the
request is empty. Correct the read timeout calculation to reflect
that.

Thanks to Stackpath for helping to debug this issue.

Fixes: #2492

The commit 01fa472647fe431d09f4873f5d0cefe8b9b88919 did not in fact
remove the effect of the cli_buffer value, but it did prevent varnish
from crashing when setting the parameter very high. This reverts the
change in the documentation.

Fixes #2468

This helps write scripts like this:

    optstring=$(varnishd -x optstring)

    while getopts $optstring opt
    do
        case $opt in
        n)
            # handle $OPTARG
            ;;
        # handle other options
        *)
            # ignore unneeded options
            ;;
        esac
    done

    varnishd "$@"

Otherwise if optstring is not kept in sync, getopts will stop processing
options if it encounters one that is not specified.

If the cli buffer is allocated on the stack, the mgt process will
crash hard for big values of this parameter. Instead we allocate on
the heap.

The new cli buffer an "auto" VSB. This means that the varnish
parameter cli_buffer is ignored from now on, and the manual marks it
as deprecated.

Closes: #2382

Conflicts:
	bin/varnishd/mgt/mgt_cli.c

This is a back port to 4.1, and the conflicts are partly a result of the
non inclusion of 656982a5 in 4.1.

(Back port by Pål Hermunn Johansen, hermunn@varnish-software.com)

Conflicts:
	bin/varnishd/cache/cache_pool.c
	bin/varnishd/cache/cache_wrk.c

Having a stack on the heap just feels unclean, also this way we have a chance
to get a red zone adjacent to the mapping just in case we manage to overflow
the alt stack also.

Ref: #2396

Ref: #2396

The locking around the use of vcl_active does not include the checking
of its magic value or the temperature asserts, leading to a race when
changing vcl_active.

Fixes: #2390

Previously, we could run out of stack handling stack overflows, leaving
users with unspecific SIGSEGV crashes and no panic message.

By providing a single alternative stack exclusively for SIGSEGV handling
where sigaltstack() is available, we increase chances for our signal handler
to finish successfully.

In particular, this will make it easier to diagnose stack overflows by
comparing the failing address with the stack info from the panic output.

This could be further improved by giving advise to increase thread_pool_stack
if si_addr is near the stack boundaries.

c00057.vtc now triggers a stack overflow instead of raising a SIGSEGV.

Merges #2396

it was already used for more than SIGSEGV, so we should output the actual
signal description.

Conflicts:
	bin/varnishd/mgt/mgt_child.c

The previous code allowed the compiler to re-read nxt from
rr->nxt which could have been incremented cocurrently.

Fixes #2378

The file stevedore may return a buffer larger than asked for when
requesting storage. Due to lack of check for this condition, the code
to copy the synthetic error memory buffer from vcl_error would overrun
the buffer.

Patch by @shamger

Fixes: #2429

The goal of `req.ttl` is to allow lower requirements for known
transactions (since this has to be done in VCL) but objects with
an age higher than `req.ttl` would always be considered expired
during lookup even if their actual TTL (obj.ttl) is still positive.

Ignoring `req.ttl` gives a better control over explicit refreshes
made to optimize latency and greatly reduces the risk of running
into #1799 (for a subset of use cases).

Refs #2422

My previous patch did not update the manual and the usage message for
varnishtest concerning the default internal buffer size.

Since SunOS needs a litt bit more workspace_thread, this test case
needs to be updated.

The default buffer size for vtc (configurable with -b) had to be
increased.

Conflicts:
	bin/varnishtest/vtc_main.c

When the do..while loop in HSH_Purge executes on a oh with many
popular variants, there is a potential problem with the "array" of oc
pointers, allocated in the thread workspace. If many of the oc's have
positive refcounts, they will fill up the array and

	EXP_Rearm(oc, now, ttl, grace, keep);
	(void)HSH_DerefObjCore(wrk, &oc, 0);

will be called several on the same oc's. At the same time, the counter
n_obj_purged will be updated with a too low number.

The test case demonstrates how we get a too low value for this counter,
but it is not able to force varnishd to use a siginificant amount of CPU.

Conflicts:
	include/tbl/oc_flags.h
	bin/varnishd/cache/cache_hash.c
	bin/varnishd/cache/cache.h

The use case are cluster requests: Intra-cluster bgfetches should
trigger a synchronous fetch on the peer-varnish in order to avoid
additional short-lived / expired objects being created. Or, in
other words, a bgfetch should actually get a fresh object and not
one in grace from another cache.

Merges #2376

This is a back port of d7f8b531.

Conflicts:
	bin/varnishd/cache/cache_fetch.c
	doc/changes.rst
	include/tbl/bo_flags.h

Fixes #2380.

Closes: #2357

Related to #2337 and #2366.

From Emmanuel Hocdet (manu-at-gandi-dot-net) via #2373.