This mostly moves out functions that are not on the fast path and
we don't try to inline. Also just clears up a lot of clutter.