[vlc-devel] OpenGL handle of RGB32 is broken

["zhilizhao(赵志立)"]

Hi all,
I found the problem with vlc screen://.

The following patch shows what caused the problem. How RGB fallback is handled
in interop_sw.c? Why interop_rgb_base_init doesn’t take account of fmt_in? Is
this still work in progress?

diff --git a/modules/video_output/opengl/interop.c b/modules/video_output/opengl/interop.c
index 96d548519e..6ded931189 100644
--- a/modules/video_output/opengl/interop.c
+++ b/modules/video_output/opengl/interop.c
@@ -321,6 +321,24 @@ interop_rgb_base_init(struct vlc_gl_interop *interop, GLenum tex_target,
             break;

         case VLC_CODEC_RGB32:
+            if (interop->fmt_in.i_rmask == 0x00ff0000
+                    && interop->fmt_in.i_gmask == 0x0000ff00
+                    && interop->fmt_in.i_bmask == 0x000000ff)
+            {
+                if (GetTexFormatSize(interop->vt, tex_target, GL_BGRA, GL_RGBA,
+                                     GL_UNSIGNED_BYTE) != 32)
+                    return VLC_EGENERIC;
+                interop->texs[0] = (struct vlc_gl_tex_cfg) {
+                    { 1, 1 }, { 1, 1 }, GL_RGBA, GL_BGRA, GL_UNSIGNED_BYTE
+                };
+            }
+            else
+            {
+                interop->texs[0] = (struct vlc_gl_tex_cfg) {
+                    { 1, 1 }, { 1, 1 }, GL_RGBA, GL_RGBA, GL_UNSIGNED_BYTE
+                };
+            }
+            break;
         case VLC_CODEC_RGBA:
             interop->texs[0] = (struct vlc_gl_tex_cfg) {
                 { 1, 1 }, { 1, 1 }, GL_RGBA, GL_RGBA, GL_UNSIGNED_BYTE

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.videolan.org/pipermail/vlc-devel/attachments/20201030/1ab1a4d8/attachment.html>