[vlc-devel] [PATCH] avcodec: update the output format when the decoder requests one

[Steve Lhomme]

We may reuse the decoder because the decoder size matches but we still need to
send other changes (like SAR) upstream otherwise we may never notice.

If the vout update fails we don't reuse the decoder and try to use a new one.

Fixes #21329
---
 modules/codec/avcodec/video.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/modules/codec/avcodec/video.c b/modules/codec/avcodec/video.c
index cccc84d22f..e972c149c1 100644
--- a/modules/codec/avcodec/video.c
+++ b/modules/codec/avcodec/video.c
@@ -1668,8 +1668,12 @@ static enum PixelFormat ffmpeg_GetFormat( AVCodecContext *p_context,
      for (size_t i = 0; pi_fmt[i] != AV_PIX_FMT_NONE; i++)
         if (pi_fmt[i] == p_sys->pix_fmt)
         {
-            msg_Dbg(p_dec, "reusing decoder output format %d", pi_fmt[i]);
-            return p_sys->pix_fmt;
+            if (lavc_UpdateVideoFormat(p_dec, p_context, p_sys->pix_fmt, swfmt) == 0)
+            {
+                msg_Dbg(p_dec, "reusing decoder output format %d", pi_fmt[i]);
+                return p_sys->pix_fmt;
+            }
+            break;
         }
 
 no_reuse:
-- 
2.17.1