[vlc-devel] [PATCH 5/5] direct3d11: fix sampler using unset memory

[Steve Lhomme]

When the decoder needs padding, that padding is not seen as pixel border by
direct3d11. So the interpolation algorithm doesn't kick in and we just use pixel
data from the padding to display pixels when resizing. In the normal case when
resizing the sampler sees that it needs a pixel outside of the texture aread
and use the algorithm selected to create fake border pixels.
See http://www.3dgep.com/texturing-lighting-directx-11/#Clamp
---
 modules/video_output/win32/direct3d11.c | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/modules/video_output/win32/direct3d11.c b/modules/video_output/win32/direct3d11.c
index 3d674fd..efb2619 100644
--- a/modules/video_output/win32/direct3d11.c
+++ b/modules/video_output/win32/direct3d11.c
@@ -1616,6 +1616,14 @@ static int Direct3D11UnmapTexture(picture_t *picture)
 {
     picture_sys_pool_t *p_sys = (picture_sys_pool_t*)picture->p_sys;
     vout_display_t     *vd = p_sys->vd;
+
+    if ( picture->p[0].i_visible_lines != picture->p[0].i_lines )
+    {
+        memset( picture->p[0].p_pixels + picture->p[0].i_pitch * (picture->p[0].i_visible_lines-1),
+                0,
+                (picture->p[0].i_lines - picture->p[0].i_visible_lines + 1) * picture->p[0].i_pitch );
+    }
+
     ID3D11DeviceContext_Unmap(vd->sys->d3dcontext, (ID3D11Resource *)p_sys->texture, 0);
     return VLC_SUCCESS;
 }
-- 
2.7.0