[vlc-commits] avcodec: update the output format when the decoder requests one

[Steve Lhomme]

vlc/vlc-3.0 | branch: master | Steve Lhomme <robux4 at ycbcr.xyz> | Tue Apr 16 13:22:57 2019 +0200| [22afd972c08bf77e81fe09b4373e8731e06165ec] | committer: Steve Lhomme

avcodec: update the output format when the decoder requests one

We may reuse the decoder because the decoder size matches but we still need to
send other changes (like SAR) upstream otherwise we may never notice.

If the vout update fails we don't reuse the decoder and try to use a new one.

Fixes #21329

> http://git.videolan.org/gitweb.cgi/vlc/vlc-3.0.git/?a=commit;h=22afd972c08bf77e81fe09b4373e8731e06165ec
---

 modules/codec/avcodec/video.c | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/modules/codec/avcodec/video.c b/modules/codec/avcodec/video.c
index 8cb4003561..097e7cb11a 100644
--- a/modules/codec/avcodec/video.c
+++ b/modules/codec/avcodec/video.c
@@ -1557,8 +1557,11 @@ static enum PixelFormat ffmpeg_GetFormat( AVCodecContext *p_context,
      for (size_t i = 0; pi_fmt[i] != AV_PIX_FMT_NONE; i++)
         if (pi_fmt[i] == p_sys->pix_fmt)
         {
-            msg_Dbg(p_dec, "reusing decoder output format %d", pi_fmt[i]);
-            return p_sys->pix_fmt;
+            if (lavc_UpdateVideoFormat(p_dec, p_context, p_sys->pix_fmt, swfmt) == 0)
+            {
+                msg_Dbg(p_dec, "reusing decoder output format %d", pi_fmt[i]);
+                return p_sys->pix_fmt;
+            }
         }
 
 no_reuse: