[PATCH] lua: rc: prevent interface from terminating VLC immediately

[Pierre Ynard]

I just discovered that last year, the cli and rc interface aliases
were removed from the lua CLI and put back on the old C RC module. I'm
surprised, and I can't say that I agree with this or that it makes
sense to me; the lua CLI was supposed to merge all available transports
(console, TCP socket, telnet) and command shells, including the VLM
shell: VLM shell commands aren't accessible from the C RC interface, so
anyone expecting them to work with the rc/cli interface will find them
missing now, and that's a regression. Another regression is the cli-host
configuration option going missing.

That being said, and this context being given, I don't understand too
well the current problem:

> When starting VLC without stdin attached (for example on Windows from
> the explorer or on macOS from Finder by double clicking the app) the
> lua interface if set up as extra interface would immediately terminate
> it again.
> For users that set the extra interface up using the GUI, they are
> no longer able to revert that change as the main interface will
> immediately terminate together with VLC, which is really bad user
> experience.

Which branch is the target of this patch? How do you even manage to
trigger such a setup, apart from editing vlcrc by hand or passing
specific lua options on the command line? The preference widgets use
the cli alias for the lua CLI, and now that this points to the C
module, how could it even launch the lua CLI? The QT interface even
has conditional code to prevent listing the lua CLI on Windows (in
preferences_widgets.cpp). And the C module will simply refuse to start
the extra interface rather than causing VLC to quit if stdin is not a
TTY, except on Windows, where, if memory serves, it will instead take
care of spawning its own Windows console.

I guess there would also be a problem when starting an extra
interface right away through the GUI rather than enabling it in the
preferences, but then again the code listing available interfaces to add
(src/interface/interface.c) uses the rc alias which now points to the C
module, and also has a check to prevent listing the CLI on Windows or if
stdin is not a TTY.

What does MacOS do when starting an application without a console,
doesn't it redirect stdin to /dev/null ? Doesn't it have isatty() and
fail that check?

Let's assume that you're running VLC 3.0, not on Windows, and managed to
enable the lua CLI. I guess it would behave like you explained.

However please note that the CLI can also be configured to accept
TCP connections, and when those close, you don't want to exit the
interface: you want to let it run and keep accepting later connections.
So the patch is wrong in that regard (and it doesn't exit the interface
properly either). Listening to stdin is only the default configuration,
and it can be disabled or coexist with TCP sockets.

Maybe a better approach would be something like adding an isatty() check
in Open_LuaCLI() similar to the one in the C module, that aborts the
interface start rather than closing VLC. This is overly restrictive in
the case where the CLI would be configured to use sockets; the C module
already presents that same issue though.

This would also break piping commands into the CLI, which was a
supported use case; so the lua CLI would have to start honoring the
rc-fake-tty option. An alternative could be to check if stdin is already
EOF when starting the interface, and then abort: if the CLI is the main
interface, the result is more or less the same as VLC will shut down if
it can't load any main interface. This couldn't be done from the lua
code though, as then it's too late to fail the module load.

-- 
Pierre Ynard
"Une âme dans un corps, c'est comme un dessin sur une feuille de papier."