[vlc-devel] [PATCH] fix some Wayland key issues when playing out videos

[erwan.tulou at gmail.com]

Hello,

    Please, find attached three patches to help fix some key issues 
preventing vlc to run smoothly on Wayland

      - patch 1: a fix to ensure video is displayed within the qt interface.

      - patch 2: a fix to video misplacement for Opengl/wayland vout 
display.

      - patch 3: a fix to sporadic crash because of null value

These patches were tested successfully both with Weston and Gnome on a 
Debian.

Erwan Tulou



---
L'absence de virus dans ce courrier électronique a été vérifiée par le logiciel antivirus Avast.
https://www.avast.com/antivirus
-------------- next part --------------
From d0f8a56b29e0fbb0e4b243b0a982575fbca6c389 Mon Sep 17 00:00:00 2001
From: Erwan Tulou <erwan10 at videolan.org>
Date: Sun, 10 Dec 2017 19:00:15 +0100
Subject: [PATCH 1/3] qt(wayland): fix video widget failing to display video.

Setting Qt:WA_DontCreateNativeAncestors in addition to Qt::WA_NativeWindow
tells qt to create a native window for the widget and only for the widget.

For Wayland, this resulted in one single wayland subsurface instead of a
tree of nested subsurfaces. Wayland is a bit convoluted when it comes to
mapping/unmapping nested subsurfaces, which accounted for the video failing
to be displayed.

For X11, on the contrary, setting Qt:WA_DontCreateNativeAncestors proved
wrong with some misplacement in the video. So, this parameter is set
*** ONLY *** in a Wayland context.
---
 modules/gui/qt/components/interface_widgets.cpp | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/modules/gui/qt/components/interface_widgets.cpp b/modules/gui/qt/components/interface_widgets.cpp
index a5fef977d3..98dfdebfbe 100644
--- a/modules/gui/qt/components/interface_widgets.cpp
+++ b/modules/gui/qt/components/interface_widgets.cpp
@@ -164,6 +164,9 @@ bool VideoWidget::request( struct vout_window_t *p_wnd )
 #ifdef QT5_HAS_WAYLAND
         case VOUT_WINDOW_TYPE_WAYLAND:
         {
+            /* Ensure only the video widget is native (needed for Wayland) */
+            stable->setAttribute( Qt::WA_DontCreateNativeAncestors, true);
+
             QWindow *window = stable->windowHandle();
             assert(window != NULL);
             window->create();
-- 
2.15.1

-------------- next part --------------
From 569eff1c2f9985a32c1c901d9198e992f07b88da Mon Sep 17 00:00:00 2001
From: Erwan Tulou <erwan10 at videolan.org>
Date: Sun, 10 Dec 2017 17:24:35 +0100
Subject: [PATCH 2/3] egl(wayland): fix vlc_gl_Resize()

This function should report changes in size only about the original display.
All subsequent video placements are managed in vout_display_opengl_Viewport().

This fixes video misplacements found when running vlc on Wayland such as :
unset DISPLAY; vlc -V gl
---
 modules/video_output/opengl/display.c | 2 +-
 modules/video_output/opengl/egl.c     | 7 +------
 2 files changed, 2 insertions(+), 7 deletions(-)

diff --git a/modules/video_output/opengl/display.c b/modules/video_output/opengl/display.c
index 2ecdd0caaa..2ace457a7d 100644
--- a/modules/video_output/opengl/display.c
+++ b/modules/video_output/opengl/display.c
@@ -253,7 +253,7 @@ static int Control (vout_display_t *vd, int query, va_list ap)
             c.align.vertical = VOUT_DISPLAY_ALIGN_TOP;
 
         vout_display_PlacePicture (&place, src, &c, false);
-        vlc_gl_Resize (sys->gl, place.width, place.height);
+        vlc_gl_Resize (sys->gl, c.display.width, c.display.height);
         if (vlc_gl_MakeCurrent (sys->gl) != VLC_SUCCESS)
             return VLC_EGENERIC;
         vout_display_opengl_SetWindowAspectRatio(sys->vgl, (float)place.width / place.height);
diff --git a/modules/video_output/opengl/egl.c b/modules/video_output/opengl/egl.c
index 9162675447..4543f2d66a 100644
--- a/modules/video_output/opengl/egl.c
+++ b/modules/video_output/opengl/egl.c
@@ -53,7 +53,6 @@ typedef struct vlc_gl_sys_t
 #endif
 #if defined (USE_PLATFORM_WAYLAND)
     struct wl_egl_window *window;
-    unsigned width, height;
 #endif
     PFNEGLCREATEIMAGEKHRPROC    eglCreateImageKHR;
     PFNEGLDESTROYIMAGEKHRPROC   eglDestroyImageKHR;
@@ -82,11 +81,7 @@ static void Resize (vlc_gl_t *gl, unsigned width, unsigned height)
 {
     vlc_gl_sys_t *sys = gl->sys;
 
-    wl_egl_window_resize(sys->window, width, height,
-                         (sys->width - width) / 2,
-                         (sys->height - height) / 2);
-    sys->width = width;
-    sys->height = height;
+    wl_egl_window_resize(sys->window, width, height, 0, 0);
 }
 #else
 # define Resize (NULL)
-- 
2.15.1

-------------- next part --------------
From 194f7ad6cf7f0e77063932e4dbc63d1d6bb84d97 Mon Sep 17 00:00:00 2001
From: Erwan Tulou <erwan10 at videolan.org>
Date: Wed, 8 Nov 2017 01:37:20 +0100
Subject: [PATCH 3/3] qt: fix width/height with zero value

In a full wayland context, these width/height may come up as equal to 0, and
this leads to vlc crashing in both the wl and the gl vout display.

Not sure though if these zero can be valid values in this context ?
The patch solved the issue without any obvious regression.
---
 modules/gui/qt/main_interface.cpp | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/modules/gui/qt/main_interface.cpp b/modules/gui/qt/main_interface.cpp
index 3b576ac23d..96591fe4f8 100644
--- a/modules/gui/qt/main_interface.cpp
+++ b/modules/gui/qt/main_interface.cpp
@@ -1322,7 +1322,8 @@ void MainInterface::resizeWindow(int w, int h)
         return;
     }
 #endif
-    resize(w, h);
+    if( w > 0 && h > 0 )
+        resize(w, h);
 }
 
 /**
-- 
2.15.1