[vlc-devel] [PATCH] Scale the picture when using YUVP, but sys->scale->p_module is nullptr

[Filipe Cabecinhas]

The problem described here happens with current git master on OS X (El
Capitan, at least).

Previously, we weren't scaling it in some cases, which will make us
access memory we shouldn't, when vout_display_opengl_Prepare (in
modules/video_output/opengl.c) passes the picture to Upload(), telling
it to use GL_RGBA format. Since the YUVP picture has 128-byte pitch for
a 128-wide picture (OSDIcon), we would error when the system eventually
called memmove with 128*4-byte lines.

Turning on AddressSanitizer makes finding this problem much easier (I
was only getting crashes on full-screen, with short jumps, but with ASan
I find memory errors any time I trigger an OSDIcon to display (windowed
mode too). I've also seen distortions on the OSDIcons every once in a
while. I'm assuming it's the same bug.

I am unsure if this 'if' condition is the most appropriate one, though.
---
 src/video_output/vout_subpictures.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/src/video_output/vout_subpictures.c b/src/video_output/vout_subpictures.c
index 196038a..0fd141c 100644
--- a/src/video_output/vout_subpictures.c
+++ b/src/video_output/vout_subpictures.c
@@ -767,10 +767,10 @@ static void SpuRenderRegion(spu_t *spu,
     }
 
     /* Scale from rendered size to destination size */
-    if (sys->scale && sys->scale->p_module &&
-        (!using_palette || (sys->scale_yuvp && sys->scale_yuvp->p_module)) &&
+    if (((sys->scale && sys->scale->p_module && !using_palette) ||
+         (sys->scale_yuvp && sys->scale_yuvp->p_module)) &&
         (scale_size.w != SCALE_UNIT || scale_size.h != SCALE_UNIT ||
-        using_palette || convert_chroma)) {
+         using_palette || convert_chroma)) {
         const unsigned dst_width  = spu_scale_w(region->fmt.i_visible_width,  scale_size);
         const unsigned dst_height = spu_scale_h(region->fmt.i_visible_height, scale_size);
 
-- 
2.6.0.rc3