[vlc-commits] [Git][videolan/vlc][master] d3d_dynamic_shader: Fix uses of out of bounds values in MultMat43

[Steve Lhomme (@robUx4)]

Steve Lhomme pushed to branch master at VideoLAN / VLC


Commits:
313d3f8e by Martin Storsjö at 2021-12-09T08:27:03+00:00
d3d_dynamic_shader: Fix uses of out of bounds values in MultMat43

Since 6ab9638604f3b2d36a6b2f928d3d1ee86e53c092, the matrices in the
function are only 4x3, but the function kept using the fourth row of
the right hand matrix.

The out of bounds values were in practice multiplied with 0, which is
why this wasn't initially noticed. However, 0 * <uninit> can also
evaluate to NaN, if the uninitialized data happens to make the bit
pattern of a NaN.

In the upcoming Clang 14, the default options for float handling
were changed slightly (in Clang commit
https://github.com/llvm/llvm-project/commit/f04e387055e495e3e14570087d68e93593cf2918).

As Clang can see that these multiplications use undefined values out of
bounds, it assumes that those values are NaNs, which let Clang essentially
optimize out most of D3D_SetupQuad, just setting all of
quad->shaderConstants->Colorspace to NaNs.

This fixes the colorspace matrix when built with Clang 14, by assuming
the fourth, missing, row of the right hand matrix in MultMat43 to be
[0, 0, 0, 1].

- - - - -


1 changed file:

- modules/video_output/win32/d3d_shaders.c


Changes:

=====================================
modules/video_output/win32/d3d_shaders.c
=====================================
@@ -289,27 +289,27 @@ static void MultMat43(FLOAT dst[4*3], const FLOAT left[4*3], const FLOAT right[4
     FLOAT z = left[0*4 + 2];
     FLOAT w = left[0*4 + 3];
     // Perform the operation on the first row
-    dst[0*4 + 0] = (right[0*4 + 0] * x) + (right[1*4 + 0] * y) + (right[2*4 + 0] * z) + (right[3*4 + 0] * w);
-    dst[0*4 + 1] = (right[0*4 + 1] * x) + (right[1*4 + 1] * y) + (right[2*4 + 1] * z) + (right[3*4 + 1] * w);
-    dst[0*4 + 2] = (right[0*4 + 2] * x) + (right[1*4 + 2] * y) + (right[2*4 + 2] * z) + (right[3*4 + 2] * w);
-    dst[0*4 + 3] = (right[0*4 + 3] * x) + (right[1*4 + 3] * y) + (right[2*4 + 3] * z) + (right[3*4 + 3] * w);
+    dst[0*4 + 0] = (right[0*4 + 0] * x) + (right[1*4 + 0] * y) + (right[2*4 + 0] * z);
+    dst[0*4 + 1] = (right[0*4 + 1] * x) + (right[1*4 + 1] * y) + (right[2*4 + 1] * z);
+    dst[0*4 + 2] = (right[0*4 + 2] * x) + (right[1*4 + 2] * y) + (right[2*4 + 2] * z);
+    dst[0*4 + 3] = (right[0*4 + 3] * x) + (right[1*4 + 3] * y) + (right[2*4 + 3] * z) + w;
     // Repeat for all the other rows
     x = left[1*4 + 0];
     y = left[1*4 + 1];
     z = left[1*4 + 2];
     w = left[1*4 + 3];
-    dst[1*4 + 0] = (right[0*4 + 0] * x) + (right[1*4 + 0] * y) + (right[2*4 + 0] * z) + (right[3*4 + 0] * w);
-    dst[1*4 + 1] = (right[0*4 + 1] * x) + (right[1*4 + 1] * y) + (right[2*4 + 1] * z) + (right[3*4 + 1] * w);
-    dst[1*4 + 2] = (right[0*4 + 2] * x) + (right[1*4 + 2] * y) + (right[2*4 + 2] * z) + (right[3*4 + 2] * w);
-    dst[1*4 + 3] = (right[0*4 + 3] * x) + (right[1*4 + 3] * y) + (right[2*4 + 3] * z) + (right[3*4 + 3] * w);
+    dst[1*4 + 0] = (right[0*4 + 0] * x) + (right[1*4 + 0] * y) + (right[2*4 + 0] * z);
+    dst[1*4 + 1] = (right[0*4 + 1] * x) + (right[1*4 + 1] * y) + (right[2*4 + 1] * z);
+    dst[1*4 + 2] = (right[0*4 + 2] * x) + (right[1*4 + 2] * y) + (right[2*4 + 2] * z);
+    dst[1*4 + 3] = (right[0*4 + 3] * x) + (right[1*4 + 3] * y) + (right[2*4 + 3] * z) + w;
     x = left[2*4 + 0];
     y = left[2*4 + 1];
     z = left[2*4 + 2];
     w = left[2*4 + 3];
-    dst[2*4 + 0] = (right[0*4 + 0] * x) + (right[1*4 + 0] * y) + (right[2*4 + 0] * z) + (right[3*4 + 0] * w);
-    dst[2*4 + 1] = (right[0*4 + 1] * x) + (right[1*4 + 1] * y) + (right[2*4 + 1] * z) + (right[3*4 + 1] * w);
-    dst[2*4 + 2] = (right[0*4 + 2] * x) + (right[1*4 + 2] * y) + (right[2*4 + 2] * z) + (right[3*4 + 2] * w);
-    dst[2*4 + 3] = (right[0*4 + 3] * x) + (right[1*4 + 3] * y) + (right[2*4 + 3] * z) + (right[3*4 + 3] * w);
+    dst[2*4 + 0] = (right[0*4 + 0] * x) + (right[1*4 + 0] * y) + (right[2*4 + 0] * z);
+    dst[2*4 + 1] = (right[0*4 + 1] * x) + (right[1*4 + 1] * y) + (right[2*4 + 1] * z);
+    dst[2*4 + 2] = (right[0*4 + 2] * x) + (right[1*4 + 2] * y) + (right[2*4 + 2] * z);
+    dst[2*4 + 3] = (right[0*4 + 3] * x) + (right[1*4 + 3] * y) + (right[2*4 + 3] * z) + w;
     // x = left[3*4 + 0];
     // y = left[3*4 + 1];
     // z = left[3*4 + 2];



View it on GitLab: https://code.videolan.org/videolan/vlc/-/commit/313d3f8ee4691f253c301df40f023e6eb63fdf6f

-- 
View it on GitLab: https://code.videolan.org/videolan/vlc/-/commit/313d3f8ee4691f253c301df40f023e6eb63fdf6f
You're receiving this email because of your account on code.videolan.org.