[vlc-commits] swscale: modify output aspect ratio if transformation is not homothetic

[Felix Abecassis]

vlc | branch: master | Felix Abecassis <felix.abecassis at gmail.com> | Wed Apr 30 15:38:25 2014 +0200| [c021c39d818a11554e5d454aee6b43df18c6e54b] | committer: Felix Abecassis

swscale: modify output aspect ratio if transformation is not homothetic

Close #10745

> http://git.videolan.org/gitweb.cgi/vlc.git/?a=commit;h=c021c39d818a11554e5d454aee6b43df18c6e54b
---

 modules/video_chroma/swscale.c |   20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/modules/video_chroma/swscale.c b/modules/video_chroma/swscale.c
index 4b763dd..6559078 100644
--- a/modules/video_chroma/swscale.c
+++ b/modules/video_chroma/swscale.c
@@ -430,6 +430,26 @@ static int Init( filter_t *p_filter )
         return VLC_EGENERIC;
     }
 
+    if (p_filter->b_allow_fmt_out_change)
+    {
+        /*
+         * If the transformation is not homothetic we must modify the
+         * aspect ratio of the output format in order to have the
+         * output picture displayed correctly and not stretched
+         * horizontally or vertically.
+         * WARNING: this is a hack, ideally this should not be needed
+         * and the vout should update its video format instead.
+         */
+        unsigned i_sar_num = p_fmti->i_sar_num * p_fmti->i_visible_width;
+        unsigned i_sar_den = p_fmti->i_sar_den * p_fmto->i_visible_width;
+        vlc_ureduce(&i_sar_num, &i_sar_den, i_sar_num, i_sar_den, 65536);
+        i_sar_num *= p_fmto->i_visible_height;
+        i_sar_den *= p_fmti->i_visible_height;
+        vlc_ureduce(&i_sar_num, &i_sar_den, i_sar_num, i_sar_den, 65536);
+        p_fmto->i_sar_num = i_sar_num;
+        p_fmto->i_sar_den = i_sar_den;
+    }
+
     p_sys->b_add_a = cfg.b_add_a;
     p_sys->b_copy = cfg.b_copy;
     p_sys->fmt_in  = *p_fmti;