[vlc-devel] [vlc-commits] avcodec: fix direct rendering with some codecs

[Rafaël Carré]

Le Sun, 11 Dec 2011 00:29:54 +0100,
Laurent Aimar <fenrir at elivagar.org> a écrit :

> On Sat, Dec 10, 2011 at 06:07:21PM -0500, Rafaël Carré wrote:
> > Le Sat, 10 Dec 2011 23:43:22 +0100,
> > Laurent Aimar <fenrir at elivagar.org> a écrit :
> > 
> > > On Sat, Dec 10, 2011 at 11:00:07PM +0100, Rafaël Carré wrote:
> > > > vlc | branch: master | Rafaël Carré <funman at videolan.org> | Sat Dec 10 16:58:35 2011 -0500| [fcc84608b67f3c3567b96604577bfdcc0de45b53] | committer: Rafaël Carré
> > > > 
> > > > avcodec: fix direct rendering with some codecs
> > > > 
> > > > Provide edge buffers when CODEC_FLAG_EMU_EDGE is set
> > > > Fix ProRes playback with direct rendering
> > > 
> > >  From what I understand of avcodec, this is wrong. We need edge only when
> > > CODEC_FLAG_EMU_EDGE isn't set while we always set it.

Reverted.

> > Hm right, EMU_EDGE shouldn't draw in this border.
> > 
> > I'm not sure why this fixes ProRes decoding when EMU_EDGE is set,
> > perhaps libav doesn't respect the flag, or perhaps video buffers
> > provide the edges already?
> 
>  Either the prores decoder doesn't respect EMU_EDGE or it expects a greater
> buffer/pitch alignment than we provide.

We use different picture sizes for vout and decoder when using direct
rendering.

vout pictures are aligned on requested width/height, but decoder
pictures width/height are modified by picture_Setup()

    /* We want V (width/height) to respect:
        (V * p_dsc->p[i].w.i_num) % p_dsc->p[i].w.i_den == 0
        (V * p_dsc->p[i].w.i_num/p_dsc->p[i].w.i_den * p_dsc->i_pixel_size) % 16 == 0
       Which is respected if you have
       V % lcm( p_dsc->p[0..planes].w.i_den * 16) == 0
    */

IIUC we modify picture width so all planes are modulo 16 (IIUC), so
when there is chroma subsampling, even if the width is modulo 16, the
chroma planes are not necessary.

The problem might be in swscale when converting from the 10bits FFmpeg
output to xvideo yuy2.

However when running in valgrind I see an invalid write in asm in the
decoding path, so it's using correctly sized picture. Perhaps another
bug though.

It's a bit fucked up but navigating between layers of decoder, libav,
and display is quite confusing :o

pointers welcome

-- 
Rafaël Carré