This implements the first bit of phks suggestion in
https://github.com/varnishcache/varnish-cache/pull/3163#issuecomment-769718143

These VPI functions implement the operations on the recursion check
bitmask for dynamic SUB calls. Strictly speaking, the VPI subs would
not need to be concerned with the methods bitmask check and the
assertion that calls originate from the right VCL, but we centralise
these aspects for simplicity.

We also add an enum to denote check failures.