[vlc-devel] [PATCH 1/2] core-vout: increase maximum format size from 8k to 32k

[Rémi Denis-Courmont]

Le perjantaina 7. heinäkuuta 2017, 17.35.03 EEST Jean-Baptiste Kempf a écrit :
> Hello,
> 
> On Fri, 7 Jul 2017, at 17:23, Rémi Denis-Courmont wrote:
> > -    if (src->i_width == 0  || src->i_width  > 8192 ||
> > 
> >> -        src->i_height == 0 || src->i_height > 8192)
> >> 
> >> +    if (src->i_width == 0  || src->i_width  > 32768 ||
> >> 
> >> +        src->i_height == 0 || src->i_height > 32768)
> >> 
> >>          return VLC_EGENERIC;
> >>      
> >>      if (src->i_sar_num <= 0 || src->i_sar_den <= 0)
> >>      
> >>          return VLC_EGENERIC;
> > 
> > You most probably can't just do that. This will most probably lead to
> > overflow in 32-bits arithmetic, and I can't see any attempt to address
> > that in the patch series.
> Are you speaking about the picture size?

There are three separate issues.

- On objet size must be strictly less than SSIZE_MAX. On 32-bits platform, an 
object size must therefore be strictly less than 2^31.

- Likewise, if the values are used in signed arithmetic, regardless of 
platform, arithmetic would overflow beyond INT_MAX. This is (almost 
everywhere) 2^31-1.

- On 64-bits platform, even if the values are used in unsigned arithmetic, 
then wrap-around will still occur above UINT_MAX, unless there is explicit 
promotion.

I assume that the maximum pixel size is 8 bytes, (e.g. aligned packed HDR 
RGB). Then maximum square dimension is a bit below 16384, due to alignment/
padding. The largest possible squared power of two is 8192... hence the 
current code.

-- 
雷米‧德尼-库尔蒙
https://www.remlab.net/