[vlc-devel] IVTC: this week: cadence detection blues

[Juha Jeronen]

Hi all,

Latest issue of IVTC news ;)

First, a short summary of the cadence detection approaches I've tried so
far:

1) The fieldwise motion detection approach (for detecting hard field
repeats), combined with conservative filtering of possibilities (i.e.
being careful not to rule them out too early) is simply incredible. It's
the most accurate of the techniques I've tried. (This was inspired by
Vektor's classic filter. My preliminary implementation of this is in the
"PATCH 5/4" version that I posted the other day. It works, but I haven't
cleaned up the code yet.)

It seems that the "yes/no" kind of motion detection is in practice much
easier to get working properly than many other things in this area.

Fieldwise detection has the special advantage that it's trivially
data-adaptive. This is because it's rare if not nonexistent that only
one field changes between valid frames. Thus, if one field changes "much
more" than the other, the less changed one can be judged a duplicate.

2) Naturally, the first approach above leads to the idea of using
similar conservative filtering on interlace scores, checking whether the
next frame should "clearly" be composed or just passed through (and
avoiding making judgements when there is no clear winner).

I tested this yesterday. This produces a cadence detector with
properties similar to the first one, but in practice it's not as
reliable. (I was hoping that it'd match better on some input...)

This one tends to lock on incorrectly sometimes, and often it just won't
lock on at all. Rarely, though, it catches some frames that the first
one doesn't.

3) Choosing the minimal interlace score triplet (my original approach)
is starting to seem a bad strategy. It works for easy cases (agreeing
with the other two cadence detectors), but at its worst it's a lottery.

It's not easily detectable whether the results are reliable, either. The
basic outlier detection is ad hoc, and the two tuning parameters for
that are one or possibly two too many.

This approach is completely different from the others above in that
there is no conservative filtering, and in the basic variant the
detection history is not used at all in helping with the new detection.
In this sense it's a more local method - either it gives results
immediately or not at all. (Often it gives incorrect results immediately...)

Still, some method to lock on quickly is needed, if there is only one
future frame available.


A reliable way to do cadence detection is to run one or more independent
detectors, rank their output, and then analyze this filtered data stream
(possibly combined out of results from different detectors) to catch a
valid sequence of three or more positions. While (and only while) such a
sequence can be found, the cadence can be considered locked on.

Reliably detecting progressive material (to know when to switch IVTC
off) seems to require in practice detecting at least three progressive
stencil positions in a row (five progressive frames in a row) to avoid
false positives. Preferably more.


And then, today's final item: I found a corner case that the filter
doesn't handle yet: improperly hard-deinterlaced telecined material.

The opening sequence to MoonPhase is like this. (The last DVD has the
text-free OP as an extra, and indeed, it's telecined and plays properly.
It seems a hard deinterlacing occurred when the English texts were added.)

The lost resolution is, well, lost, but I think the characteristic
stutter resulting from the duplicate frames can in principle be fixed in
a fairly simple manner. So maybe a future version of the filter should
do this.

I did some preliminary tests, full-frame diffing the *output* from the
filter, too (including the frames that are eventually dropped).
Telecined material, including the improperly deinterlaced kind, shows
duplicates exactly every five frames (in cases with true 24fps motion,
i.e. when the stutter matters). Thus, it should be possible - after one
stutter (which is unavoidable without a five-frame future buffer) - to
start correcting the timestamps and to drop the duplicates. When a
duplicate is no longer found at exactly the fifth frame (comes earlier,
later or not at all), this logic can be switched off.

The first test version sort of works, smoothing out the stutter, but to
do this properly I need to refactor the cadence detection logic.

Also, the test version sometimes produces interlacing artifacts (by
composing the wrong field combination) when the input has none. This may
have something to do with the comb metric, so I'm not sure if it's fixable.

That's it for today.


Ah, right - the copious printf() abuse club would like to announce that
MoonPhase seems to contain some 30fps camera pans. (Interlacing is
removed correctly, but there are no full-frame output duplicates, as
there would be at 24fps.) Grh. No one was supposed to do that after 2000
or so. Here I was, thinking that there was a problem in the filter when
the pans didn't look smooth ;)

 -J