[vlc-devel] Summary of VLC and unicode
rdenis at simphalempin.com
Wed Nov 15 17:04:34 CET 2006
It's over a year than libVLC was switched to UTF-8 internally, and it's
now getting kind of useable again. Still, there's been quite many
changes since the last summary, so let's make a new one.
As a general rules, all character strings passed to LibVLC should be in
UTF-8. This includes but is far from limited to the playlist, and all
the "chain" from the input/access to the outputs. We pretty much needed
Unicode anyway; the only options were UTF-8 and wide char. We poke
UTF-8 because, as disruptive as it has been, it remains much less
disruptive than wide characters. Also wide characters means different
things on different platforms (e.g. UCS2LE on old Windows, UTF-16LE on
modern Windows, UTF-32 on glibc), are a pain to use in POSIX
environment (including Mac OS X and Linux which most of the devs are
using), and might have needed lots of libc replacements wc* functions
on some platforms.
The original approach has been to rely on ToLocale, FromLocale and
LocaleFree (and ToLocaleDup, FromLocaleDup and free) whenever a
character string in the OS native representation was needed. It turned
out to be a big mistake on Windows, which uses no less than three
different representations internally: UTF-16 in the kernel and file
system (Unicode/wide), ACP (ANSI Code Page) which is a locale-dependant
8-bit character set (CP1252 in the West), and even "OEM" (cp437 in the
US, cp850 in Western Europe). Unfortunately, (To|From)Locale were using
ACP, which prevents usage of Unicode code points outside the range of
the local charset, while the filesystem and native Win32 supports them.
This means VLC cannot "reach" some files.
So we now a comprehensive (but not fully yet) set of higher-level
wrappers that accept UTF-8 character string to represent filenames, or
text to be outputted to the console. At the time of writing, trunk
* utf8_open, utf8_fopen, utf8_stat, utf8_lstat, utf8_mkdir,
utf8_opendir, utf8_readdir, utf8_scandir for file system operation,
* utf8_fprintf, utf8_vfprintf for console output (or actually output in
any file who needs local charset).
All of these functions behave like their non "utf8_"-prefixed equivalent
with the exceptions that they expect UTF-8 character strings.
We also have EnsureUTF8 that force a string into UTF-8 (replaces invalid
character sequences with '?') and IsUTF8 that checks if the string is
valid UTF-8 (returns NULL if not). These are useful when reading UTF-8
from untrusted sources, such as the network, before injecting the data
ToLocale and FromLocale should NEVER be used for filesystem operations
anymore, except on code that is not aimed at Windows. And even then
it's better to use wrappers; for instance, Mac OS X has some conversion
logic for its directory listing. When using the Win32 API manually,
always use the Unicode version (the one ending with 'W' instead
of 'A'). If you need to convert to UTF-8 for use in LibVLC, you can use
FromWide (Win32 only!). Of course, it's better to resort to the
wrappers otherwise you always need a specific Windows implementations.
Please refer to the various pieces of code using Win32
SHFolderGetPath() for reference implementations.
Now for the good (?) news: in trunk, the only parts that still uses the
*Locale VLC API are:
- ncurses (if you are brave, you can try porting to wide ncurses...)
- GnomeVFS (which is not aimed at Win32 anyway)
- GnuTLS (not support from GnuTLS library :( )
- WxWidgets (only on non-Win32)
I have however only checked the core and plugins, so there might be
problems with browser plugins and bindings. Also, if you intend to
write new plugins, please keep this in mind.
In addition to this, I remind you that we now have the
GetFallbackEncoding() API that returns the ANSI code page used by
Windows for the given locale, to be passed to iconv. For instance, if
your system is configured to run in German, it returns "CP1252". This
is convenient when handling subtitles or various kind of texts that are
typically aimed at Windows users.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Size: 189 bytes
Desc: not available
More information about the vlc-devel