Also fixes a memory leak where the side data was previously not properly
cleaned up on OOM.
Signed-off-by: Anton Khirnov <anton@khirnov.net>