[vlc-devel] VLC RBG chroma issues

[basos g]

VLC has some issues with RBG chromas that i am trying to fix.
This problem is not always present (on all RGB chromas) but avutil to
VLC chroma mapping is broken and works with reverse and again reverse
logic for some chromas. This could be observed when for example
swscale converts from RV24 to some YcBcR chroma where red and blue are
interchanged.

The misunderstanding is about how do we handle endianess.

Some RBG chromas are handled natively (by avutil, swscale) concerning
their endianness
RV32, RV16, RV15 (that are very common ) are some of them.
RV24 is not natively handled for example.

RGB chroma endianess is clued either :
a) by current arch (when no mask info is present), which seems logical to me
b) by mask information : eg we migh occure RV24
0xFF0000,0x00FF00,0x0000FF or RV24 0x0000FF, 0x00FF00, 0x0000FF

But there are scatered vlc code parts where RBG masks are set with
kindof arbitrary way for the big endian arch.
My question is should rgb masks carry the correct endianess
information ( depending on arch ) or should they have always the big
endian values and let the compiler do the magic ? For this to be
ansered i have to understand where mask information are used appart
from VLC to avutil mappings.

I have done some work on this. But we have to decide what msk
information (e.g. carried on the picture format) should mean.
My suggestion is to modify every affecting part (easily grepable) to
carry the endianess-aware mask info.