Commit 18f24527 accidentally made side data only packets be handled like a
flush request. Fix this regression by effectively ignoring them as was the
original intention.
Signed-off-by: James Almer <jamrial@gmail.com>