[vlc-commits] [Git][videolan/vlc][master] 3 commits: swscale: fix OOB read when scaling a cropped alpha plane
Steve Lhomme (@robUx4)
gitlab at videolan.org
Sat Jul 4 15:45:22 UTC 2026
Steve Lhomme pushed to branch master at VideoLAN / VLC
Commits:
c6d12114 by Thomas Guillem at 2026-07-04T15:17:30+00:00
swscale: fix OOB read when scaling a cropped alpha plane
The alpha plane is extracted into a visible-sized GREY picture
(p_src_a), but the shared Convert() re-applied the source crop offset to
it. With a cropped input (e.g. an aspect-ratio thumbnail), sws_scale
started reading p_src_a at that offset and ran lines past the buffer:
SIGSEGV on Windows.
Scale the extracted plane from its own origin via a dedicated ConvertA()
instead of routing it through Convert(), which is meant for the full-size
colour pictures.
Command:
```
./bin/vlc-preparser-static --verbose 2 alpha_codecPNGImage.mov \
--type thumbnail_to_files --seek-pos 0.1 --output-path out.rgba \
--output-width 128 --output-height 64 --output-format rgba --output-crop
```
gdb backtrace on Windows:
```
[11:39:19] "VLC [level 0]: 1920x960 (1920x1080) chroma: RGBA colorspace: BT709 -> 128x64 (128x64) chroma: RGBA colorspace: BT709 with scaling using Bicubic (good quality)"
warning: main generic debug: using video converter module "swscale"
[11:39:19] "VLC [level 0]: using video converter module \"swscale\""
Thread 44 "vlc-run-thumb" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 15320.0x2e10]
0x00007ffbff38c616 in ff_hscale8to15_X4_avx2 () from /c/Users/videolabs/Desktop/tguillem/vlc-4.0.0-dev/plugins/video_chroma/libswscale_plugin.dll
(gdb) bt
```
Valgrind log on Linux:
```
[0000000004e9b170] swscale filter debug: 1920x960 (1920x1080) chroma: RGBA colorspace: BT709 -> 128x64 (128x64) chroma: RGBA colorspace: BT709 with scaling using Bicubic (good quality)
[00000000068e69d0] main generic debug: using video converter module "swscale"
==3471261== Thread 3 vlc-run-thfil:
==3471261== Invalid read of size 8
==3471261== at 0x714A01A: ??? (in /home/tom/work/git/vlc/build/modules/.libs/libswscale_plugin.so)
==3471261== by 0x385: ???
==3471261== by 0x71574BD: lum_h_scale (hscale.c:57)
==3471261== by 0x6FD056E: ff_swscale (swscale.c:501)
==3471261== by 0x6FD1A20: scale_internal (swscale.c:1179)
==3471261== by 0x6FD2A16: sws_scale (swscale.c:1535)
==3471261== by 0x6FA8AC9: Convert (swscale.c:719)
==3471261== by 0x6FA9E67: Filter (swscale.c:787)
==3471261== by 0x4957674: ImageWrite (image.c:431)
==3471261== by 0x49500FC: picture_Export (picture.c:648)
==3471261== by 0x48D59A5: ThumbnailerToFilesRun (internal.c:451)
==3471261== by 0x494A38D: ThreadRun (executor.c:134)
==3471261== Address 0x73eafff is in a rw- mapped file /memfd:vlc-memfd (deleted) segment
==3471261==
==3471261==
==3471261== Process terminating with default action of signal 11 (SIGSEGV)
==3471261== Access not within mapped region at address 0x73EB000
==3471261== at 0x714A01A: ??? (in /home/tom/work/git/vlc/build/modules/.libs/libswscale_plugin.so)
==3471261== by 0x385: ???
==3471261== by 0x71574BD: lum_h_scale (hscale.c:57)
==3471261== by 0x6FD056E: ff_swscale (swscale.c:501)
==3471261== by 0x6FD1A20: scale_internal (swscale.c:1179)
==3471261== by 0x6FD2A16: sws_scale (swscale.c:1535)
==3471261== by 0x6FA8AC9: Convert (swscale.c:719)
==3471261== by 0x6FA9E67: Filter (swscale.c:787)
==3471261== by 0x4957674: ImageWrite (image.c:431)
==3471261== by 0x49500FC: picture_Export (picture.c:648)
==3471261== by 0x48D59A5: ThumbnailerToFilesRun (internal.c:451)
==3471261== by 0x494A38D: ThreadRun (executor.c:134)
```
- - - - -
bba0df11 by Thomas Guillem at 2026-07-04T15:17:30+00:00
swscale: extract the alpha plane from the cropped origin
ExtractA() read the source alpha from (0,0), grabbing the uncropped
top-left region, while the colour plane is scaled from the crop offset.
The rescaled alpha therefore did not line up with the colour for cropped
input. Start the extraction (and the planar-alpha copy) at the crop
origin so the alpha matches the colour plane.
- - - - -
5519a21d by Thomas Guillem at 2026-07-04T15:17:30+00:00
test: add run_alpha_crop_test
This test was failing before the commit: "swscale: fix OOB read when
scaling a cropped alpha plane"
```
Run alpha crop test with internal preparser...
[00007fcce0020370] avcodec encoder error: Unknown option "border_mask"
[00007fcce0020370] avcodec encoder error: Unknown option "noise_reduction"
test_src_preparser_thumbnail_to_files: ../../test/src/preparser/thumbnail_to_files.c:480: run_alpha_crop_test: Assertion `buf[i * 4 + 3] == buf[i * 4 + 0] && "swscale alpha-crop regression: alpha does not match colour"' failed.
```
This test was failing before the commit: "swscale: extract the alpha
plane from the cropped origin"
```
Run alpha align test with internal preparser...
[00007f67b80333d0] avcodec encoder error: Unknown option "border_mask"
[00007f67b80333d0] avcodec encoder error: Unknown option "noise_reduction"
test_src_preparser_thumbnail_to_files: ../../test/src/preparser/thumbnail_to_files.c:468: run_alpha_crop_test: Assertion `max_alpha > 200 && "swscale alpha-crop regression: alpha extracted from wrong origin"' failed.
```
- - - - -
2 changed files:
- modules/video_chroma/swscale.c
- test/src/preparser/thumbnail_to_files.c
Changes:
=====================================
modules/video_chroma/swscale.c
=====================================
@@ -606,14 +606,16 @@ static void GetPixels( uint8_t *pp_pixel[4], int pi_pitch[4],
}
static void ExtractA( picture_t *p_dst, const picture_t *restrict p_src,
- unsigned offset )
+ unsigned x_offset, unsigned y_offset, unsigned offset )
{
plane_t *d = &p_dst->p[0];
const plane_t *s = &p_src->p[0];
+ const uint8_t *src = s->p_pixels + y_offset * s->i_pitch
+ + x_offset * s->i_pixel_pitch + offset;
for( unsigned y = 0; y < p_dst->format.i_height; y++ )
for( unsigned x = 0; x < p_dst->format.i_width; x++ )
- d->p_pixels[y*d->i_pitch+x] = s->p_pixels[y*s->i_pitch+4*x+offset];
+ d->p_pixels[y*d->i_pitch+x] = src[y*s->i_pitch+x*s->i_pixel_pitch];
}
static void InjectA( picture_t *p_dst, const picture_t *restrict p_src,
@@ -634,6 +636,20 @@ static void FillA( plane_t *d, unsigned i_offset )
d->p_pixels[y*d->i_pitch+x+i_offset] = 0xff;
}
+static void ConvertA( struct SwsContext *ctx, plane_t *dst_plane,
+ const plane_t *src_plane, int i_height )
+{
+ /* src/dst are GREY planes that already hold the cropped,
+ * de-offset alpha. Feed swscale their raw pixels. */
+ const uint8_t *src[4] = { src_plane->p_pixels, NULL, NULL, NULL };
+ uint8_t *dst[4] = { dst_plane->p_pixels, NULL, NULL, NULL };
+ const int src_stride[4] = { src_plane->i_pitch, 0, 0, 0 };
+ const int dst_stride[4] = { dst_plane->i_pitch, 0, 0, 0 };
+
+ sws_scale( ctx, src, src_stride, 0, i_height, dst, dst_stride );
+}
+
+
static void CopyPad( picture_t *p_dst, const picture_t *p_src )
{
picture_Copy( p_dst, p_src );
@@ -769,14 +785,19 @@ static picture_t *Filter( filter_t *p_filter, picture_t *p_pic )
{
/* We extract the A plane to rescale it, and then we reinject it. */
if( p_fmti->i_chroma == VLC_CODEC_RGBA || p_fmti->i_chroma == VLC_CODEC_BGRA )
- ExtractA( p_sys->p_src_a, p_src, OFFSET_A );
+ ExtractA( p_sys->p_src_a, p_src, p_fmti->i_x_offset, p_fmti->i_y_offset, OFFSET_A );
else if( p_fmti->i_chroma == VLC_CODEC_ARGB || p_fmti->i_chroma == VLC_CODEC_ABGR )
- ExtractA( p_sys->p_src_a, p_src, 0 );
+ ExtractA( p_sys->p_src_a, p_src, p_fmti->i_x_offset, p_fmti->i_y_offset, 0 );
else
- plane_CopyPixels( p_sys->p_src_a->p, p_src->p+A_PLANE );
+ {
+ plane_t a = p_src->p[A_PLANE];
+ a.p_pixels += p_fmti->i_y_offset * a.i_pitch
+ + p_fmti->i_x_offset * a.i_pixel_pitch;
+ plane_CopyPixels( p_sys->p_src_a->p, &a );
+ }
- Convert( p_filter, p_sys->ctxA, p_sys->p_dst_a, p_sys->p_src_a,
- p_fmti->i_visible_height, 1, false, false );
+ ConvertA( p_sys->ctxA, &p_sys->p_dst_a->p[0], &p_sys->p_src_a->p[0],
+ p_fmti->i_visible_height );
if( p_fmto->i_chroma == VLC_CODEC_RGBA || p_fmto->i_chroma == VLC_CODEC_BGRA )
InjectA( p_dst, p_sys->p_dst_a, OFFSET_A );
else if( p_fmto->i_chroma == VLC_CODEC_ARGB || p_fmto->i_chroma == VLC_CODEC_ABGR )
=====================================
test/src/preparser/thumbnail_to_files.c
=====================================
@@ -28,7 +28,10 @@
#include <vlc_modules.h>
#include <vlc_fs.h>
+#include <stdint.h>
+#include <stdio.h>
#include <errno.h>
+#include <unistd.h>
#define MOCK_WIDTH 640
#define MOCK_HEIGHT 480
@@ -349,6 +352,144 @@ skip:
return 77;
}
+struct alpha_context
+{
+ vlc_sem_t sem;
+ bool success;
+};
+
+static void alpha_on_ended(vlc_preparser_req *req, int status,
+ const bool *result_array, size_t result_count,
+ void *data)
+{
+ struct alpha_context *ctx = data;
+ (void) req;
+ assert(status == VLC_SUCCESS);
+ assert(result_count == 1);
+ ctx->success = result_array[0];
+ vlc_sem_post(&ctx->sem);
+}
+
+/* Regression test for:
+ * - test_align=false: "swscale: fix OOB read when scaling a cropped alpha plane"
+ * - test_align=true: "swscale: extract the alpha plane from the cropped origin"
+ */
+static int run_alpha_crop_test(libvlc_instance_t *vlc, bool test_align)
+{
+ /* Needs an RGBA image encoder (avcodec); skip otherwise. */
+ if (vlc_preparser_CheckThumbnailerFormat(VLC_THUMBNAILER_FORMAT_RGBA) != 0)
+ {
+ fprintf(stderr, "skip: no RGBA thumbnailer format\n");
+ return 0;
+ }
+
+ char path[] = "/tmp/libvlc_XXXXXX";
+ int fd = vlc_mkstemp(path);
+ if (fd == -1)
+ {
+ fprintf(stderr, "skip: vlc_mkstemp failed\n");
+ return 0;
+ }
+
+ /* 256x128 (2:1) from a 640x480 (4:3) source forces a non-zero centered crop
+ * (640x320, y_offset 80), which the colorbar alignment check below relies
+ * on to keep the bottom opaque band. */
+ const unsigned ow = 256, oh = 128;
+ const struct vlc_thumbnailer_output output = {
+ .format = VLC_THUMBNAILER_FORMAT_RGBA,
+ .width = ow, .height = oh, .crop = true,
+ .file_path = path, .creat_mode = 0666,
+ };
+
+ const struct vlc_preparser_cfg cfg = {
+ .types = VLC_PREPARSER_TYPE_THUMBNAIL_TO_FILES,
+ .max_parser_threads = 1,
+ .max_thumbnailer_threads = 1,
+ .timeout = 0,
+ .external_process = false,
+ };
+ vlc_preparser_t *preparser = vlc_preparser_New(VLC_OBJECT(vlc->p_libvlc_int),
+ &cfg);
+ assert(preparser != NULL);
+
+ struct alpha_context ctx = { .success = false };
+ vlc_sem_init(&ctx.sem, 0);
+
+ const struct vlc_thumbnailer_arg arg = {
+ .seek = { .type = VLC_THUMBNAILER_SEEK_POS, .pos = 0.2,
+ .speed = VLC_THUMBNAILER_SEEK_FAST },
+ .hw_dec = false,
+ };
+ static const struct vlc_thumbnailer_to_files_cbs cbs = {
+ .on_ended = alpha_on_ended,
+ };
+
+ /* RGBA source so the conversion exercises swscale's alpha path. */
+ input_item_t *item = input_item_New(
+ test_align ? MOCK_URL ";video_chroma=RGBA;video_colorbar=true"
+ : MOCK_URL ";video_chroma=RGBA", "mock");
+ assert(item != NULL);
+
+ vlc_preparser_req *req =
+ vlc_preparser_GenerateThumbnailToFiles(preparser, item, &arg, &output, 1,
+ &cbs, &ctx);
+ assert(req != NULL);
+ vlc_sem_wait(&ctx.sem);
+
+ vlc_preparser_req_Release(req);
+ input_item_Release(item);
+ vlc_preparser_Delete(preparser);
+
+ assert(ctx.success);
+
+
+ const size_t size = (size_t)ow * oh * 4;
+ uint8_t *buf = malloc(size);
+ assert(buf != NULL);
+ assert(lseek(fd, 0, SEEK_SET) == 0);
+ for (size_t off = 0; off < size; )
+ {
+ ssize_t r = read(fd, buf + off, size - off);
+ assert(r > 0);
+ off += r;
+ }
+ if (test_align)
+ {
+ /* With the mock colorbar, a 640x480 RGBA frame has 4 horizontal bands
+ * of 120 lines; only the bottom band (rows 360..479) is opaque (alpha
+ * 0xFF), the rest is transparent. A 2:1 output crops to 640x320
+ * centered (rows 80..399), so the cropped region includes the bottom
+ * opaque band. If the alpha is (wrongly) extracted from (0,0), it
+ * covers rows 0..319 which are fully transparent, and the output alpha
+ * is 0 everywhere. */
+ uint8_t max_alpha = 0;
+ for (size_t i = 0; i < (size_t)ow * oh; i++)
+ if (buf[i * 4 + 3] > max_alpha)
+ max_alpha = buf[i * 4 + 3];
+ /* Loose midpoint: the band is kept (alpha ~0xFF) or dropped (alpha 0).
+ * The exact scaled value is implementation-dependent. */
+ assert(max_alpha > 200
+ && "swscale alpha-crop regression: alpha extracted from wrong origin");
+ }
+ else
+ {
+ /* Read back the raw RGBA. The mock fills every byte with the same
+ * value, so R == G == B == A on input. The colour planes go through
+ * the in-bounds main rescale, the alpha through the cropped alpha
+ * path, so on a correct conversion the output alpha still equals the
+ * red channel everywhere. The out-of-bounds bug leaves the alpha
+ * garbage (it differs from red). */
+ for (size_t i = 0; i < (size_t)ow * oh; i++)
+ assert(buf[i * 4 + 3] == buf[i * 4 + 0]
+ && "swscale alpha-crop regression: alpha does not match colour");
+ }
+ free(buf);
+
+ close(fd);
+ unlink(path);
+ return 0;
+}
+
int main(int argc, const char *argv[])
{
test_init();
@@ -372,6 +513,16 @@ int main(int argc, const char *argv[])
if (ret != 0) {
goto end;
}
+ fprintf(stderr, "Run alpha crop test with internal preparser...\n");
+ ret = run_alpha_crop_test(vlc, false);
+ if (ret != 0) {
+ goto end;
+ }
+ fprintf(stderr, "Run alpha align test with internal preparser...\n");
+ ret = run_alpha_crop_test(vlc, true);
+ if (ret != 0) {
+ goto end;
+ }
#if !defined(HAVE_VLC_PROCESS_SPAWN)
fprintf(stderr, "Run with external preparser...\n");
ret = run_test(vlc, true);
View it on GitLab: https://code.videolan.org/videolan/vlc/-/compare/5cd300ad532fdc98f0b36e6cd7f6891e6a94e3f8...5519a21dc1cc968963a2c74b03b2718c0a3216c5
--
View it on GitLab: https://code.videolan.org/videolan/vlc/-/compare/5cd300ad532fdc98f0b36e6cd7f6891e6a94e3f8...5519a21dc1cc968963a2c74b03b2718c0a3216c5
You're receiving this email because of your account on code.videolan.org. Manage all notifications: https://code.videolan.org/-/profile/notifications | Help: https://code.videolan.org/help
More information about the vlc-commits
mailing list