[vlc-devel] [PATCH] core: do not overwrite visible size

[Felix Abecassis]

2013/11/9 Rémi Denis-Courmont <remi at remlab.net>
>
> Libavcodec often requires an extra block of lines at the bottom for optimized direct rendering. That is probably why.

Yes, avcodec_align_dimensions2 aligns width to a multiple of 16 and
height to a multiple of 32 + 2 (apparently). The avcodec module checks
the dimensions of the allocated pictures it receives, and if it is
smaller than the dimensions returned by this function then direct
rendering is disabled.

The above patch seems to be required since the visible dimensions
should not be overwritten.
However this does not solve the bug because swscale is using the
"hidden" resolution and not the visible one. During scaling we are
therefore interpolating new pixels using uninitialized values from the
extra lines added by avcodec at the bottom.

Modifying the swscale module to have it work on visible dimensions
(for input/output) is simple but the thumbnails still have garbage
lines at the bottom. There is yet another bug; swscale is modifying
the visible dimensions of the output image using the function
video_format_ScaleCropAr. The output image is modified in order to
have the same hidden to visible ratio between the input and the output
pictures for both width and height:
p_dst->i_visible_width  = (uint64_t)p_src->i_visible_width  *
p_dst->i_width  / p_src->i_width;
p_dst->i_visible_height = (uint64_t)p_src->i_visible_height *
p_dst->i_height / p_src->i_height;

Why do that? That doesn't make sense.
For instance, with an input 48x48 video we have:
Input: 48x66 48x48
Output: 800x450 800x327
We now have 48/66 ~= 327/450 but we are not getting the resolution we
asked for. The thumbnailer (for instance) will read a 800x450 picture
from the buffer.
It looks like the call to video_format_ScaleCropAr should be removed here.

With all the changes mentioned above, it looks like I don't have green
lines anymore on thumbnails and videos. I will do more extensive
testing and send a patch next week.
Also, I couldn't reproduce the issue reported by XilasZ on 720p videos.

Feedbacks welcomed.

-- 
Félix Abecassis
http://felix.abecassis.me