[vlc-devel] [PATCH] swscale: Fix pixel format for VLC_CODEC_RGBA

[Laurent Aimar]

On Thu, Dec 20, 2012 at 03:02:47PM +0100, Florian Albrechtskirchinger wrote:
> On 12/20/2012 02:43 PM, Laurent Aimar wrote:
>> On Thu, Dec 20, 2012 at 02:29:48PM +0100, falbrechtskirchinger at gmail.com wrote:
>>> From: Florian Albrechtskirchinger <falbrechtskirchinger at gmail.com>
>>>
>>> Hi,
>>>
>>> this patch resolves inverted reds and blues during XCB screen capture of a semi-transparent window.
>>> The issue was introduced by commit fb7f014161f68c914e028adbea68b7ee9863c76e, which changed the pixel format from PIX_FMT_RGBA32 to PIX_FMT_BGR32.
>>   I am not sure that it is right. I fear that the issue is in the XCB module
>> that declares RGBA but returns data that does not follow the order that
>> VLC RGBA wants (RGBA is only supported with a fixed order in VLC).
>>
>> A solution/workaround may be to return RGB32 in the XCB plugin.
>>
>> I will let courmisch comments.
>>
> I did look at the XCB plugin initially and wrote some code to convert  
> the pixels based on the red/green/blue_masks in the xcb_visualtype_t  
> structure, only to realize, that there is nothing to convert. The masks  
> are identical for RGBA and RGB24 (which is selected for non-transparent  
> windows, and doesn't exhibit inverted colors).
> It took me a while, but eventually I figured out the culprit is  
> PIX_FMT_BGR32 in the swscale wrapper.
I am unsure, I would need to double check, RGBA is used in subtitles rendering
and there the color is right.

> So what order is VLC expecting? It seems to be 0xAARRGGBB, right?
RGBA is: R at byte 0, G at byte 1, B at byte 2, A at byte 3.

-- 
fenrir