This a backport of f9aa6281 in master.

Fixes #1665.

Fixes #1462

Spotted by: gquintard

Fixes: #1637

Fixes: #1672

Conflicts:
	bin/varnishd/cache/cache_fetch.c

Conflicts:
	bin/varnishd/cache/cache_fetch.c

This state could be set by the expiry timer even though a reference is
grabbed during lookup, causing the assertion to trigger.

Fixes: #1539

The real issue was a missing reference/, which the additional
requirement should take care of.

Note to future self: install -D is something entirely different
on BSD systems.

tri-graphs.

Fixes #1566

Conflicts:
	doc/sphinx/Makefile.am

I will have to retag 4.0.3-rc2 because of this.

Not tagging it yet.

Conflicts:
	bin/varnishd/cache/cache_http.c
	bin/varnishd/http1/cache_http1_fsm.c

Fixes #1616

Conflicts:
	varnish.m4

Add missing bits and sync with reality. Relevant fixes:

* a synth object is never cached
* by a backend_response object may be

Fixes #1479

Fixes #1616

Conflicts:
	bin/varnishd/Makefile.am
	varnish.m4

This updates the last oustanding document to 4.0

Conflicts:
	doc/sphinx/users-guide/operation-logging.rst

This could happen on a 200 synth object if the client request had a
Range header. The range code path would fail because it requires an
obj which the synth path doesn't have.

Fixes: #1660

Keep the fetch thread busyobj pointer-ref separate from the client thread pointer-ref, and deref it on thread scheduling failure.

The code tried to deref the same pointer twice, which failed because
the VBO_DerefBusyobj() will clear the pointer when called. Separating
allows calling VBO_DerefBusyobj() for each of them.

Fixes: #1628
(cherry picked from commit 7746e30e)

Conflicts:
	bin/varnishd/cache/cache_fetch.c

Don't attempt to run the fetch in the request thread if there are no threads available in the pool, fail the fetch and count it.

Fixes #1620
Fixes #1621

(cherry picked from commit 969674dc)

If we fail to reschedule from the waiting list, we ditch the entire waiting list in an attempt to recover.

This also fixes buglet in previous commit which failed to call
CNT_AcctLogCharge()

Fixes: #1629

(cherry picked from commit 9fbcb4e9)

Per our usual naming scheme, use -rc1 instead of -beta1.

Fixes: #1648

Conflicts:
	bin/varnishd/cache/cache_fetch.c

Continuously update our total object size estimate when cond-fetching a still-being-streamed object.

Fixes:	#1646

Keeping only the test case as the issue isn't in 4.0.

Conflicts:
	bin/varnishd/cache/cache_fetch.c

Fixes #1638
Fixes #1641

Ported to 4.0 by Martin

Conflicts:
	bin/varnishd/cache/cache_fetch.c