[vlc-commits] vout: fix overflow in subpicture sar

[Francois Cartegnie]

vlc/vlc-2.2 | branch: master | Francois Cartegnie <fcvlcdev at free.fr> | Sat Oct 21 20:54:13 2017 +0200| [51066c31f8d582051bfd752a9c4870f1d3c86e91] | committer: Francois Cartegnie

vout: fix overflow in subpicture sar

triggers incorrect sar / subpicture size when
sar is set (scaling down)

backport of c2f5a6fce003fcfb0fda3fb0fcda28ec0d801e6a

> http://git.videolan.org/gitweb.cgi/vlc/vlc-2.2.git/?a=commit;h=51066c31f8d582051bfd752a9c4870f1d3c86e91
---

 src/video_output/vout_subpictures.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/src/video_output/vout_subpictures.c b/src/video_output/vout_subpictures.c
index a051c16b41..646dc84b06 100644
--- a/src/video_output/vout_subpictures.c
+++ b/src/video_output/vout_subpictures.c
@@ -1051,10 +1051,14 @@ static subpicture_t *SpuRenderSubpictures(spu_t *spu,
             /* Compute region scale AR */
             video_format_t region_fmt = region->fmt;
             if (region_fmt.i_sar_num <= 0 || region_fmt.i_sar_den <= 0) {
-                region_fmt.i_sar_num = (int64_t)fmt_dst->i_visible_width  * fmt_dst->i_sar_num * subpic->i_original_picture_height;
-                region_fmt.i_sar_den = (int64_t)fmt_dst->i_visible_height * fmt_dst->i_sar_den * subpic->i_original_picture_width;
+
+                const uint64_t i_sar_num = (uint64_t)fmt_dst->i_visible_width  *
+                                           fmt_dst->i_sar_num * subpic->i_original_picture_height;
+                const uint64_t i_sar_den = (uint64_t)fmt_dst->i_visible_height *
+                                           fmt_dst->i_sar_den * subpic->i_original_picture_width;
+
                 vlc_ureduce(&region_fmt.i_sar_num, &region_fmt.i_sar_den,
-                            region_fmt.i_sar_num, region_fmt.i_sar_den, 65536);
+                            i_sar_num, i_sar_den, 65536);
             }
 
             /* Compute scaling from original size to destination size