[vlc-devel] Cropping

[Steve Lhomme]

I was looking at the cropping handling in D3D11 with offset_test.ogv
and noticed many issues. Although I managed to place the rectangle
correctly to only show the inside square
(http://v2v.cc/~j/theora_testsuite/offset_test.pass.png) the bottom of
the picture was green, suggesting a part of the area was not filled at
all. So I pre-filled the picture with a color and indeed, that area is
not filled by the decoder.

I then tried with D3D9 by forcing RGBA support, so that swscale is
used and it even crashes. It seems that swscale is trying to write in
unallocated memory. As it turns out, D3D9 allocates only an area of
the visible size, not of the decoder size. Even with I420 enabled,
D3D9 fails to display offset_test.ogv correctly.

>From my understanding when a picture_t has an i_width/height,
i_x/y_offset and i_visible_width/height, the planes should be
allocated with width/height but only the visible area should be used
and the offset handled to display the right position of that visible
rectangle. Am I correct ? That means the current way of D3D9 to deal
with it is wrong.

Also is the decoder supposed to decode the whole area
(http://v2v.cc/~j/theora_testsuite/offset_test.fail.png) or it can
take shortcuts to fill only the area it knows will be visible ? A
bogus shortcut could explain why I'm missing some areas filled with
D3D11 and the right decoding area (right number of visible lines
decoded but the vertical offset not applied).

If the vout cannot handle cropping by itself, is a crop filter
inserted automatically in the vout filter chain ? If so, how does the
vout signal if it can handle cropping or not.