av_display_rotation_get will return NAN when the display matrix is invalid,
which would end up printing NAN as an integer in the rotation field. This
is poor for multiple reasons:

* Users of ffprobe have no way of discerning "valid but ugly rotation from
  display matrix" from "invalid display matrix".
* It can have unintended consequences on some platforms, such as Linux x86_64,
  where NAN is equal to INT64_MIN, which, for example, when printed as JSON,
  which uses floating point for all numbers, can end up as invalid JSON or wit
  a number that cannot be reserialized as an integer at all.

Since NAN is av_display_rotation_get's error case, just print 0 (no rotation)
when that happens.
Signed-off-by: Derek Buitenhuis <derek.buitenhuis@gmail.com>