[vlc-devel] [PATCH] core: fix vlc_alloc() overflow detection

[Romain Vimont]

The overflow detection failed to detect some multiplication overflows.

For example:

    #include <stdint.h>
    #include <stdio.h>

    int main() {
        size_t count = SIZE_MAX / 2 + 3;
        size_t size = 4;
        size_t mul = count * size;
        printf("       count = %zu\n", count);
        printf("        size = %zu\n", size);
        printf("count * size = %zu\n", mul);
        int accept_old = count * size >= size;
        int accept_new = size && count <= SIZE_MAX / size;
        printf("overflow detected (old): %d\n", !accept_old);
        printf("overflow detected (new): %d\n", !accept_new);
        return 0;
    }

On my computer, this program prints:

           count = 9223372036854775810
            size = 4
    count * size = 8
    overflow detected (old): 0
    overflow detected (new): 1

See <http://www.informit.com/articles/article.aspx?p=1959565&seqNum=13>.

Note that the new condition is false when size is 0, even if there is no
overflow. In that case, directly returning NULL is ok.

Signed-off-by: Romain Vimont <rom at rom1v.com>
---
 include/vlc_common.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/vlc_common.h b/include/vlc_common.h
index ed5ff77730..d860d31bd9 100644
--- a/include/vlc_common.h
+++ b/include/vlc_common.h
@@ -847,7 +847,7 @@ VLC_API bool vlc_ureduce( unsigned *, unsigned *, uint64_t, uint64_t, uint64_t )
 VLC_USED VLC_MALLOC
 static inline void *vlc_alloc(size_t count, size_t size)
 {
-    return likely(count * size >= size) ? malloc(count * size) : NULL;
+    return likely(size && count <= SIZE_MAX / size) ? malloc(count * size) : NULL;
 }
 
 /*****************************************************************************
-- 
2.11.0