[vlc-devel] [RFC] avcodec: copy frame-based palette

[Daniel Verkamp]

On Tue, May 13, 2014 at 10:08 PM, Daniel Verkamp <daniel at drv.nu> wrote:
> ---
>  modules/codec/avcodec/video.c  | 13 +++++++++++++
>  modules/video_chroma/swscale.c |  2 ++
>  2 files changed, 15 insertions(+)

With this change, codecs with per-frame palettes now display correctly
(e.g. Smacker - many more unmapped codecs from libavcodec depend on
this too).

However, I am not sure if the change is semantically correct - see below.

> diff --git a/modules/codec/avcodec/video.c b/modules/codec/avcodec/video.c
> index 2e119d0..3e47cab 100644
> --- a/modules/codec/avcodec/video.c
> +++ b/modules/codec/avcodec/video.c
> @@ -890,6 +890,19 @@ static void ffmpeg_CopyPicture( decoder_t *p_dec,
>          uint8_t *p_dst, *p_src;
>          int i_src_stride, i_dst_stride;
>
> +        if( p_ff_pic->palette_has_changed )
> +        {
> +            if( !p_pic->format.p_palette )
> +            {
> +                p_pic->format.p_palette = calloc( 1, sizeof(video_palette_t) );
> +                p_pic->format.p_palette->i_entries = 256;
> +            }
> +            if( p_pic->format.p_palette )
> +            {
> +                memcpy( p_pic->format.p_palette->palette, p_ff_pic->data[1], AVPALETTE_SIZE );
> +            }
> +        }
> +

The palette is stored in the picture_t of the frame where it is
updated here when a palette change occurs.

> diff --git a/modules/video_chroma/swscale.c b/modules/video_chroma/swscale.c
> index 2763e43..3b2981d 100644
> --- a/modules/video_chroma/swscale.c
> +++ b/modules/video_chroma/swscale.c
> @@ -584,6 +584,8 @@ static void Convert( filter_t *p_filter, struct SwsContext *ctx,
>      if( p_filter->fmt_in.video.i_chroma == VLC_CODEC_RGBP )
>      {
>          memset( palette, 0, sizeof(palette) );
> +        if( p_src->format.p_palette )
> +            p_filter->fmt_in.video.p_palette = p_src->format.p_palette;
>          if( p_filter->fmt_in.video.p_palette )
>              memcpy( palette, p_filter->fmt_in.video.p_palette->palette,
>                      __MIN( sizeof(video_palette_t), AVPALETTE_SIZE ) );

Later frames don't have the palette, so I copy the video_palette_t
pointer into the filter input format in swscale.  This seems a little
ugly to me; perhaps there is a better place to store the palette
between palette changes.

I am also not sure where the palette is actually freed, so the palette
pointer may not be safe to copy into the filter.

Thanks,
-- Daniel