[vlc-devel] [PATCH] swscale: Fix pixel format for VLC_CODEC_RGBA

[Florian Albrechtskirchinger]

On 12/20/2012 02:43 PM, Laurent Aimar wrote:
> On Thu, Dec 20, 2012 at 02:29:48PM +0100, falbrechtskirchinger at gmail.com wrote:
>> From: Florian Albrechtskirchinger <falbrechtskirchinger at gmail.com>
>>
>> Hi,
>>
>> this patch resolves inverted reds and blues during XCB screen capture of a semi-transparent window.
>> The issue was introduced by commit fb7f014161f68c914e028adbea68b7ee9863c76e, which changed the pixel format from PIX_FMT_RGBA32 to PIX_FMT_BGR32.
>   I am not sure that it is right. I fear that the issue is in the XCB module
> that declares RGBA but returns data that does not follow the order that
> VLC RGBA wants (RGBA is only supported with a fixed order in VLC).
>
> A solution/workaround may be to return RGB32 in the XCB plugin.
>
> I will let courmisch comments.
>
I did look at the XCB plugin initially and wrote some code to convert 
the pixels based on the red/green/blue_masks in the xcb_visualtype_t 
structure, only to realize, that there is nothing to convert. The masks 
are identical for RGBA and RGB24 (which is selected for non-transparent 
windows, and doesn't exhibit inverted colors).
It took me a while, but eventually I figured out the culprit is 
PIX_FMT_BGR32 in the swscale wrapper.
So what order is VLC expecting? It seems to be 0xAARRGGBB, right?

Florian