Move some of the code out into a separate source and header.

This mostly moves out functions that are not on the fast path and
we don't try to inline. Also just clears up a lot of clutter.