[vlc-commits] [Git][videolan/vlc][master] omxil: fix memory leak on format convertion

[Hugo Beauzée-Luyssen (@chouquette)]

Hugo Beauzée-Luyssen pushed to branch master at VideoLAN / VLC


Commits:
91b19f2b by Maxim Romanov at 2022-07-26T06:58:21+00:00
omxil: fix memory leak on format convertion

After 8f42dc5c08826514ae33a42e5da28ff098f4383c commit
h264_avcC_to_AnnexB_NAL() and hevc_hvcC_to_AnnexB_NAL() are always
allocate memory using malloc(). Allocated block assigned to pBuffer
and changed initially allocated pointer value. According to OpenMAX
specification, OMX component should not free this buffer and it will
leak.

Furthermore, OMX component may rely on initially allocated buffer
address.

Copying the data into the original buffer and free temporary memory
block should fix both issues.

- - - - -


1 changed file:

- modules/codec/omxil/omxil.c


Changes:

=====================================
modules/codec/omxil/omxil.c
=====================================
@@ -1037,18 +1037,52 @@ static int OpenGeneric( vlc_object_t *p_this, bool b_encode )
             h264_isavcC(p_dec->fmt_in.p_extra, p_dec->fmt_in.i_extra) )
         {
             size_t i_filled_len = 0;
-            p_header->pBuffer = h264_avcC_to_AnnexB_NAL(
+            uint8_t *p_buf = h264_avcC_to_AnnexB_NAL(
                         p_dec->fmt_in.p_extra, p_dec->fmt_in.i_extra,
                         &i_filled_len, NULL );
+
+            if( p_buf == NULL )
+            {
+                msg_Dbg(p_dec, "h264_avcC_to_AnnexB_NAL() failed");
+                goto error;
+            }
+
+            if( i_filled_len > p_header->nAllocLen )
+            {
+                msg_Dbg(p_dec, "buffer too small (%i,%i)", (int)i_filled_len,
+                        (int)p_header->nAllocLen);
+                free(p_buf);
+                goto error;
+            }
+
+            memcpy(p_header->pBuffer, p_buf, i_filled_len);
             p_header->nFilledLen = i_filled_len;
+            free(p_buf);
         }
         else if( p_dec->fmt_in.i_codec == VLC_CODEC_HEVC && !p_sys->in.b_direct )
         {
             size_t i_filled_len = 0;
-            p_header->pBuffer = hevc_hvcC_to_AnnexB_NAL(
+            uint8_t *p_buf = hevc_hvcC_to_AnnexB_NAL(
                         p_dec->fmt_in.p_extra, p_dec->fmt_in.i_extra,
                         &i_filled_len, &p_sys->i_nal_size_length );
+
+            if( p_buf == NULL )
+            {
+                msg_Dbg(p_dec, "hevc_hvcC_to_AnnexB_NAL() failed");
+                goto error;
+            }
+
+            if( i_filled_len > p_header->nAllocLen )
+            {
+                msg_Dbg(p_dec, "buffer too small (%i,%i)", (int)i_filled_len,
+                        (int)p_header->nAllocLen);
+                free(p_buf);
+                goto error;
+            }
+
+            memcpy(p_header->pBuffer, p_buf, i_filled_len);
             p_header->nFilledLen = i_filled_len;
+            free(p_buf);
         }
         else if(p_sys->in.b_direct)
         {



View it on GitLab: https://code.videolan.org/videolan/vlc/-/commit/91b19f2b025a71c9f92bb09813ca973433c1919a

-- 
View it on GitLab: https://code.videolan.org/videolan/vlc/-/commit/91b19f2b025a71c9f92bb09813ca973433c1919a
You're receiving this email because of your account on code.videolan.org.


VideoLAN code repository instance