[vlc-devel] [PATCH] core: fix vlc_obj_alloc_common() overflow detection

[Romain Vimont]

The overflow detection failed to detect some multiplication overflows.

For example:

    #include <stdint.h>
    #include <stdio.h>

    int main() {
        size_t nmemb = 4;
        size_t size = SIZE_MAX / 2 + 3;
        size_t tabsize = nmemb * size;
        printf("       nmemb = %zu\n", nmemb);
        printf("        size = %zu\n", size);
        printf("nmemb * size = %zu\n", tabsize);
        int detect_old = tabsize < nmemb;
        int detect_new = size && tabsize / size != nmemb;
        printf("overflow detected (old): %d\n", detect_old);
        printf("overflow detected (new): %d\n", detect_new);
        return 0;
    }

On my computer, this program prints:

           nmemb = 4
            size = 9223372036854775810
    nmemb * size = 8
    overflow detected (old): 0
    overflow detected (new): 1

See <http://www.informit.com/articles/article.aspx?p=1959565&seqNum=13>.

Note that whenever size or nmemb is 0, the condition is false, so errno
is not set to ENOMEM.

Signed-off-by: Romain Vimont <rom at rom1v.com>
---
 src/misc/objres.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/misc/objres.c b/src/misc/objres.c
index 7645cc2274..5b6f2d2cf0 100644
--- a/src/misc/objres.c
+++ b/src/misc/objres.c
@@ -138,7 +138,7 @@ static void *vlc_obj_alloc_common(vlc_object_t *obj, size_t nmemb, size_t size,
 {
     size_t tabsize = nmemb * size;
 
-    if (unlikely(tabsize < nmemb))
+    if (unlikely(size && tabsize / size != nmemb))
     {
         errno = ENOMEM;
         return NULL;
-- 
2.11.0