[vlc-devel] [PATCH] core: do not overwrite visible size

[Felix Abecassis]

In your patch:
+    msg_Err(p_filter, "i_height=%i", i_height);
+    plane_t *p = &(p_src->p[0]);
+    msg_Err(p_filter, "src %ix%i (%ix%i)",
+            p->i_pitch / p->i_pixel_pitch, p->i_lines,
+            p->i_visible_pitch / p->i_pixel_pitch, p->i_visible_lines);
+    p = &(p_dst->p[0]);
+    msg_Err(p_filter, "dst %ix%i (%ix%i)",
+            p->i_pitch / p->i_pixel_pitch, p->i_lines,
+            p->i_visible_pitch / p->i_pixel_pitch, p->i_visible_lines);

Why are you printing these values? Picture allocation seems to have
its own padding code. In picture_Setup: (src/misc/picture.c)
const int i_width_aligned  = ( i_width  + i_modulo_w - 1 ) /
i_modulo_w * i_modulo_w;
const int i_height_aligned = ( i_height + i_modulo_h - 1 ) /
i_modulo_h * i_modulo_h;
const int i_height_extra   = 2 * i_ratio_h; /* This one is a hack for
some ASM functions */

Where i_modulo_w and i_modulo_h are computed using the chroma type of
the picture.

So we have two levels of padding in the code:
- Plane allocation: i_visible_pitch/i_visible_lines vs
i_pitch/i_lines. Because of the above code, avcodec DR is still
enabled even if we don't call avcodec_align_dimensions2 (but we
shouldn't rely on that obviously).
- Picture: i_visible_width/i_visible_height vs i_width/i_height. Using
also i_x_offset/i_y_offset this can be used to implement cropping. But
it's also used for padding (avcodec/Theora modules).
Since it looks like we cannot influence the alignment used during
plane allocation we have to use this instead.

In function Convert, p_src.format and p_dst.format seems to be
consistent with the resolution swscale was configured with.