Restores behavior from before 20cacfe4.
Eventually this should be handled similarly to audio/video - with a
filtergraph reset or a manual scaler.