We still own it on failure, and there's no point trying to feed it again.

This should address the issue reported in dav1d #383 and part of VLC #26259.
Signed-off-by: James Almer <jamrial@gmail.com>
Reviewed-by: Ronald S. Bultje <rsbultje@gmail.com>