[vlc-devel] [Alexander Gall <gall at switch.ch>] Re: Fwd: [Possible endian bug in video_chrome/video_output]

[System administration]

----- Forwarded message from Alexander Gall <gall at switch.ch> -----

From: Alexander Gall <gall at switch.ch>
Date: Mon, 3 Apr 2006 15:47:21 +0200
To: Derk-Jan Hartman <hartman at videolan.org>
Cc: vlc-devel at videolan.org
Subject: Re: [vlc-devel] Fwd: [Possible endian bug in video_chrome/video_output]
X-Mailer: VM 7.18 under Emacs 21.3.3
X-Spam-Status: No, score=-5.7 required=5.0 tests=GAPPY_TEXT,IN_REP_TO 
	autolearn=failed version=3.0.3

The deleted attachment is at:
    <http://www.videolan.org/~admin/20060404-videolan/pjointe>
 
On Sat, 1 Apr 2006 01:27:49 +0200, Derk-Jan Hartman <hartman at videolan.org> said:

> Could you please check the values that are used in

> #define RGB2PIXEL( p_vout, i_r, i_g, i_b )          \
>      (((((uint32_t)i_r) >> p_vout->output.i_rrshift) \
>                         << p_vout->output.i_lrshift) \
>     | ((((uint32_t)i_g) >> p_vout->output.i_rgshift) \
>                         << p_vout->output.i_lgshift) \
>     | ((((uint32_t)i_b) >> p_vout->output.i_rbshift) \
>                         << p_vout->output.i_lbshift))

> on your machine. Sam suspects your problem is in those values.

I have 

i_rmask: 0x000000ff
i_rrshift: 0
i_lrshift: 0

i_gmask: 0x0000ff00
i_rrshift: 0
i_lrshift: 8

i_bmask: 0x00ff0000
i_rrshift: 0
i_lrshift: 16

The mask values are wrong.  They should be

i_rmask: 0xff000000
i_gmask: 0x00ff0000
i_bmask: 0x0000ff00

I'm not sure where exactly these are set for opengl.  If it's in
modules/video_output/opengl.c, then the problem is obvious, because
the values are just set statically, instead of read from the
capabilities of the output device.

Now, when I set the proper mask values by hand, the output is still
not ok.  This is because src/video_output/video_output.c:MaskToShift()
produces the wrong result for a mask of 0xff000000.  The line

    i_high = i_mask + i_low;                       /* higher bit of the mask */

results in an overflow and the value of rshift becomes 31369 (it
should be 0).

When I fix this as well, the output is finally correct.

BTW, MaskToShift seems to expect that the mask can be smaller than 8
bits, in which case the lower order bits of a color value are
discarded by RGB2PIXEL.  I'm wondering whether this makes sense and if
it actually occurs in practice.

Finally, the same problem needs to be fixed for the xvideo and glx
output modules as well.

--
Alex

----- End forwarded message -----

-- 
System administration <admin at via.ecp.fr>
VIA, École Centrale Paris, France

-- 
This is the vlc-devel mailing-list, see http://www.videolan.org/vlc/
To unsubscribe, please read http://developers.videolan.org/lists.html