[vlc-devel] [PATCH v2 03/17] opengl: simplify renderer format

[Romain Vimont]

The interop may modify its own copy of the video_format_t. These changes
must be reported to the core (by writing to the provided
video_format_t).

But as a hack, the Android interop currently flips the orientation to
avoid a vertical flip in the renderer, since it provides its own
transform matrix. This change must not be reported to the core (the core
must not flip the input), so the orientation change is reverted. A
further refactor will remove this hack later.

Meanwhile, handle the format to report to the core outside of the
renderer, so that the renderer is not involved in this hack. This paves
the way to pass only the interop to the renderer, without an additional
video_format_t instance.
---
 modules/video_output/opengl/renderer.c    | 5 +----
 modules/video_output/opengl/vout_helper.c | 7 ++++++-
 2 files changed, 7 insertions(+), 5 deletions(-)

diff --git a/modules/video_output/opengl/renderer.c b/modules/video_output/opengl/renderer.c
index 828ffb5621a..a2f321a6692 100644
--- a/modules/video_output/opengl/renderer.c
+++ b/modules/video_output/opengl/renderer.c
@@ -495,10 +495,7 @@ vlc_gl_renderer_New(vlc_gl_t *gl, const struct vlc_gl_api *api,
                                   sampler->var.OrientationMatrix);
     getViewpointMatrixes(renderer, interop->fmt.projection_mode);
 
-    /* Update the fmt to main program one */
-    renderer->fmt = interop->fmt;
-    /* The orientation is handled by the orientation matrix */
-    renderer->fmt.orientation = fmt->orientation;
+    renderer->fmt = *fmt;
 
     /* Texture size */
     for (unsigned j = 0; j < interop->tex_count; j++) {
diff --git a/modules/video_output/opengl/vout_helper.c b/modules/video_output/opengl/vout_helper.c
index 1945f01e715..31b828d64a8 100644
--- a/modules/video_output/opengl/vout_helper.c
+++ b/modules/video_output/opengl/vout_helper.c
@@ -187,7 +187,12 @@ vout_display_opengl_t *vout_display_opengl_New(video_format_t *fmt,
         return NULL;
     }
 
-    *fmt = renderer->fmt;
+    video_orientation_t orientation = fmt->orientation;
+    *fmt = vgl->interop->fmt;
+    /* The orientation is handled by the orientation matrix: from the core
+     * point of view, it does not change. */
+    fmt->orientation = orientation;
+
     if (subpicture_chromas) {
         *subpicture_chromas = gl_subpicture_chromas;
     }
-- 
2.26.2