[vlc-devel] commit: Remove UTF-32 hacks from stream. ( Rémi Denis-Courmont )

[Rémi Denis-Courmont]

On Mon, 1 Sep 2008 11:30:33 +0200, Derk-Jan Hartman <hartman at videolan.org>
wrote:
> On 31 aug 2008, at 20:33, git version control wrote:
>> vlc | branch: master | Rémi Denis-Courmont <rdenis at simphalempin.com>  
>> | Sun Aug 31 21:27:52 2008 +0300|  
>> [32f3116178b58222e28d258312141149d5b8573e] | committer: Rémi Denis- 
>> Courmont
>>
>> Remove UTF-32 hacks from stream.
>>
>> For a start, nobody uses UTF-32 for transmission/storage.
>> Then, we anyway have iconv() support in the subtitle support (where it
>> belongs IMHO), if you really wanted to use UTF-32.
> 
> Just as a thought... We could consider leaving the byte checks in  
> place and sounding off a warning in case we DO see the 32 BOM. Then at  
> least we know what happens in such a case, and we won't have to wonder  
> what bug we are hunting if it does ever occur.

*Nobody* uses UTF-32 for transmission. I mean nobody. A fortiori, nobody
encodes the BOM into UTF-32. And if someone used UTF-32 for transmission,
it is not obvious that they would use the BOM either. Asians prefer their
local character set, or UTF-16. Westerners prefer their 8-bits charsets, or
UTF-8. To use UTF-32 would only make sense if a large proportion of
characters were outside the BMP, and even then, it would slightly
under-perform UTF-16 in terms of storage space.

Also, IMHO the UCS-2 support within stream_ReadLine is a failure-prone
hack. In particular, it cannot read UCS-2 without BOM.

-- 
Rémi Denis-Courmont