The last reference to a director might go away with VRT_DelDirector
_or_ VRT_Asssign_Backend, which the former needs to account for.

We assert for the VDIR_FLG_NOREFCNT case that there was only one
reference such that a single deref yields no reference left.

Part one of the fix for #3895