[vlc-devel] [PATCH] swscale: Fix pixel format for VLC_CODEC_RGBA

[Florian Albrechtskirchinger]

On 12/20/2012 05:28 PM, Laurent Aimar wrote:
> On Thu, Dec 20, 2012 at 03:02:47PM +0100, Florian Albrechtskirchinger wrote:
>> On 12/20/2012 02:43 PM, Laurent Aimar wrote:
>>> On Thu, Dec 20, 2012 at 02:29:48PM +0100, falbrechtskirchinger at gmail.com wrote:
>>>> From: Florian Albrechtskirchinger <falbrechtskirchinger at gmail.com>
>>>>
>>>> Hi,
>>>>
>>>> this patch resolves inverted reds and blues during XCB screen capture of a semi-transparent window.
>>>> The issue was introduced by commit fb7f014161f68c914e028adbea68b7ee9863c76e, which changed the pixel format from PIX_FMT_RGBA32 to PIX_FMT_BGR32.
>>>    I am not sure that it is right. I fear that the issue is in the XCB module
>>> that declares RGBA but returns data that does not follow the order that
>>> VLC RGBA wants (RGBA is only supported with a fixed order in VLC).
>>>
>>> A solution/workaround may be to return RGB32 in the XCB plugin.
>>>
>>> I will let courmisch comments.
>>>
>> I did look at the XCB plugin initially and wrote some code to convert
>> the pixels based on the red/green/blue_masks in the xcb_visualtype_t
>> structure, only to realize, that there is nothing to convert. The masks
>> are identical for RGBA and RGB24 (which is selected for non-transparent
>> windows, and doesn't exhibit inverted colors).
>> It took me a while, but eventually I figured out the culprit is
>> PIX_FMT_BGR32 in the swscale wrapper.
> I am unsure, I would need to double check, RGBA is used in subtitles rendering
> and there the color is right.
>
>> So what order is VLC expecting? It seems to be 0xAARRGGBB, right?
> RGBA is: R at byte 0, G at byte 1, B at byte 2, A at byte 3.
>
To reiterate:
On my system the screen capture chroma is either RGB32 or RGBA, 
depending on the window. In both cases Blue is at byte 0 and Red at byte 2.
By your definition the chromas should be BGR32 and BGRA respectively. 
Rémi already expressed his support of them. Therefore may I suggest 
adding them and possibly BGR24/16/15? If there are no objections, I'd 
like to give it a go.

Florian