[vlc-devel] RGB chroma mess

[Francois Cartegnie]

Hi,

I'm trying for too long now to figure out what is wrong with the RGB,
and especially RGB32 paths. (avi been fixed)

Because we inherited the x11 and bitmap headers masks there's apparently
lots if issue due to mishandling, and also setting RV32 everywhere.

My test setup:
- swscale and scale are disabled
- rv32 converter is modified to overwrite all samples and output a
single color on the RGB quad, according to the mask, and I load an RV24
sample.
- tuning the alpha channel also allows to verify if the expected chroma
contains one.

>From there, vout:

* OpenGL seems to request BGRA instead of XRGB.
-                tc->fmt.i_rmask  = 0x000000ff;
-                tc->fmt.i_gmask  = 0x0000ff00;
-                tc->fmt.i_bmask  = 0x00ff0000;
+                tc->fmt.i_rmask  = 0x0000ff00;
+                tc->fmt.i_gmask  = 0x00ff0000;
+                tc->fmt.i_bmask  = 0xff000000;
and it also requires proper alpha value.

* X11 requests XRGB and sets the proper masks, but displays it as BGR...

* None behaves the same sending the forged samples if swscale is in the
loop. (seems lots of "fixes" there).

Unsure what to do.

-- 
Francois Cartegnie
VideoLAN - VLC Developer