Up until now, it has returned the AVFilterFormats list via
an AVFilterFormats** parameter; the actual return value was an int
that was always AVERROR(ENOMEM) on error. The AVFilterFormats**
argument was a pure output parameter which was only documented
by naming the parameter rfmts. Yet nevertheless all callers
initialized the underlying AVFilterFormats* to NULL.

This commit changes this to return a pointer to AVFilterFormats
directly. This is more in line with the API in general, as it
allows to avoid checks for intermediate values.
Reviewed-by: Nicolas George <george@nsup.org>
Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>