[vlc-devel] full screen overlay for automatic projection mapping

[Alexandre Janniaux]

Hi,

On Mon, Oct 07, 2019 at 08:38:16AM +0200, Steve Lhomme wrote:
> Hi,
>
> On 2019-10-04 1:38, Andras Szalai wrote:
> > I am trying to develop a filter plugin to fix some projection issues for
> > my home cinema setup. I got as far as writing and compiling a simple
> > out-of-tree video filter plugin that modifies the video by moving some
> > pixels around.
> >
> > Eventually I want to be able to do an automatic calibration. The filter
> > would work in tandem
> > with a companion smartphone app. The filter plugin would render some
> > specific patterns on the full screen output, the smartphone app would
> > analyze the patterns and communicate back to the plugin somehow. This
> > would be great for correcting for wobbly projection screens.
>
> That sounds like a great project.
>

I agree 100%, it is really cool. It could bring a basis for
doing autocalibration with any number of video projector.

What do you get after doing the calibration and image
registration ? The projector parameters and a bump map of
the screen ?

If so maybe you'd like to execute this filter in a graphics
accelerated pipeline ?

> > The problem I am having is that filter plugins work on the video, not on
> > the output, so I don't have a way to render to the "black areas".
> >
> > I want to avoid writing an output plugin as that would be a lot more
> > work and would be platform dependent.
> >
> > My video filter plugin can potentially resize the video to match the
> > size of the output, but this would be inefficient and everything
> > downstream of the plugin would be working on a larger resolution video.
>
> I think in the current state of VLC a filter cannot really resize it's
> output, in the end a filter will be added to get back to the original size
> of the source.
>
> When we have push (a couple of weeks away) things will change. You'll be
> able to add a filter to change the output size and all following filters (if
> any) will use that as the input/output size, and that's how it will be
> displayed.
>
> IMO for your use case there's no other option than output the size of your
> projector with the black bars added on your side. That's the only way you
> can guarantee what you're modifying matches exactly your display physical
> properties.
>

I also think that you don't have the choice regarding the
output.

We've been working on extending the graphics pipeline for
this kind of use case. This could be used directly in the
video output without rewriting another display so it could
match with what you're trying to achieve. The blocking
points here would be the current statut of this work and
the way you'd like to interact with this plugin.

> > I am new to VLC internals. Is there a way to render an overlay (e.g. a
> > checkerboard pattern) from a filter plugin such that it potentially
> > covers the entire full screen output?
>
> It might be doable with the logo plugin, the logo being your pattern of the
> size of the display and semi transparent.

I fear that it could get stuck on the video canvas with the
logo filter, shifting it below the top black bar.

Could this task be handled by a mock-like pattern:// demux?
I needed this a few week ago for a similar use case and
actually used an image instead with --play-and-pause, but
this didn't allow for easy and advanced configuration, and
it wasn't standalone in the software.

Maybe there are missing parts here, as you probably want
the pattern to have a known aspect ratio while taking all
the rendering space during autocalibration and registration.

Do you have more details about your implementation ?

Regards,
--
Alexandre Janniaux
Videolabs