[vlc-devel] [PATCH 3/3] swscale: scale the aspect-ratio of the output format

[Felix Abecassis]

For instance, when using swscale after a 90 degrees rotation on 1080p,
the transformation is 1080x1920 -> 1920x1080. The aspect-ratio must be
modified in order to have the displayed image correctly rotated and
not stretched.

Fix #10745.
---
 modules/video_chroma/swscale.c | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/modules/video_chroma/swscale.c b/modules/video_chroma/swscale.c
index 11a8ecb..a22e608 100644
--- a/modules/video_chroma/swscale.c
+++ b/modules/video_chroma/swscale.c
@@ -441,6 +441,16 @@ static int Init( filter_t *p_filter )
     p_sys->b_swap_uvi = cfg.b_swap_uvi;
     p_sys->b_swap_uvo = cfg.b_swap_uvo;
 
+    /* Compute aspect-ratio of the output format. */
+    p_fmto->i_sar_num *= p_fmti->i_visible_width;
+    p_fmto->i_sar_den *= p_fmto->i_visible_width;
+    vlc_ureduce(&p_fmto->i_sar_num, &p_fmto->i_sar_den,
+                p_fmto->i_sar_num, p_fmto->i_sar_den, 65536);
+    p_fmto->i_sar_num *= p_fmto->i_visible_height;
+    p_fmto->i_sar_den *= p_fmti->i_visible_height;
+    vlc_ureduce(&p_fmto->i_sar_num, &p_fmto->i_sar_den,
+                p_fmto->i_sar_num, p_fmto->i_sar_den, 65536);
+
 #if 0
     msg_Dbg( p_filter, "%ix%i (%ix%i) chroma: %4.4s -> %ix%i (%ix%i) chroma: %4.4s extend by %d",
              p_fmti->i_visible_width, p_fmti->i_visible_height, p_fmti->i_width, p_fmti->i_height, (char *)&p_fmti->i_chroma,
-- 
1.8.3.2