[vlc-commits] dxva2: select the favorite decoder format later

[Steve Lhomme]

vlc | branch: master | Steve Lhomme <robux4 at ycbcr.xyz> | Wed Mar 11 15:01:58 2020 +0100| [5f1a0aa70cdfb8fc74fb8d099086c12c4a41716a] | committer: Steve Lhomme

dxva2: select the favorite decoder format later

We need to know the chroma subsampling and bitdepth before we can choose a good
candidate. The list of supported formats provided by the decoder may not be the
optimal choice, not respecting the minimum bitdepth or adding more chroma
subsampling if falling back to NV12.

> http://git.videolan.org/gitweb.cgi/vlc.git/?a=commit;h=5f1a0aa70cdfb8fc74fb8d099086c12c4a41716a
---

 modules/codec/avcodec/dxva2.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/modules/codec/avcodec/dxva2.c b/modules/codec/avcodec/dxva2.c
index 3dda2c6c03..6009fe826a 100644
--- a/modules/codec/avcodec/dxva2.c
+++ b/modules/codec/avcodec/dxva2.c
@@ -281,11 +281,6 @@ static int Open(vlc_va_t *va, AVCodecContext *ctx, enum PixelFormat hwfmt, const
 
     va->sys = sys;
 
-    if (desc->comp[0].depth > 8)
-        sys->render = MAKEFOURCC('P','0','1','0');
-    else
-        sys->render =  MAKEFOURCC('N','V','1','2');
-
     /* Load dll*/
     sys->dxva2_dll = LoadLibrary(TEXT("DXVA2.DLL"));
     if (!sys->dxva2_dll) {
@@ -470,6 +465,14 @@ static int DxSetupOutput(vlc_va_t *va, const directx_va_mode_t *mode, const vide
         }
     }
 
+    D3DFORMAT preferredOutput;
+    if (mode->bit_depth > 8)
+        preferredOutput = MAKEFOURCC('P','0','1','0');
+    else
+        preferredOutput =  MAKEFOURCC('N','V','1','2');
+    msg_Dbg(va, "favor decoder format %4.4s (for %d bits)", (const char*)&preferredOutput,
+            mode->bit_depth);
+
     /* */
     for (unsigned pass = 0; pass < 2 && err != VLC_SUCCESS; ++pass)
     {
@@ -483,7 +486,7 @@ static int DxSetupOutput(vlc_va_t *va, const directx_va_mode_t *mode, const vide
             }
             if (!is_supported)
                 continue;
-            if (pass == 0 && format->format != sys->render)
+            if (pass == 0 && format->format != preferredOutput)
                 continue;
 
             /* We have our solution */