[vlc-devel] [PATCH] core: fix vlc_alloc() overflow detection

[Rémi Denis-Courmont]

On dimanche 12 novembre 2017 00:10:27 EET Romain Vimont wrote:
> Note that the new condition is false when size is 0, even if there is no
> overflow. In that case, directly returning NULL is ok.

Yes, the formula works fine with addition, not with multiplication.

> Signed-off-by: Romain Vimont <rom at rom1v.com>
> ---
>  include/vlc_common.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/include/vlc_common.h b/include/vlc_common.h
> index ed5ff77730..d860d31bd9 100644
> --- a/include/vlc_common.h
> +++ b/include/vlc_common.h
> @@ -847,7 +847,7 @@ VLC_API bool vlc_ureduce( unsigned *, unsigned *,
> uint64_t, uint64_t, uint64_t ) VLC_USED VLC_MALLOC
>  static inline void *vlc_alloc(size_t count, size_t size)
>  {
> -    return likely(count * size >= size) ? malloc(count * size) : NULL;
> +    return likely(size && count <= SIZE_MAX / size) ? malloc(count * size)
> : NULL; }

It should be size == 0 || ...

>  /**************************************************************************
> ***

-- 
Rémi Denis-Courmont