Don't panic in fini it init was never called

Ref: https://github.com/varnishcache/varnish-cache/issues/2297