[x265] [PATCH] Quant: fix for decoding hash mismatch in lossless mode

[Steve Borho]

On 08/12, Ashok Kumar Mishra wrote:
> On Wed, Aug 12, 2015 at 1:05 PM, Steve Borho <steve at borho.org> wrote:
> 
> > On 08/11, ashok at multicorewareinc.com wrote:
> > > # HG changeset patch
> > > # User Ashok Kumar Mishra<ashok at multicorewareinc.com>
> > > # Date 1439283627 -19800
> > > #      Tue Aug 11 14:30:27 2015 +0530
> > > # Node ID 854f0f9aabf7ec45da3cb301e0f78a5bd1e96391
> > > # Parent  3eb2ec5922be1cd934dec7f7ed886d03c0125ef5
> > > Quant: fix for decoding hash mismatch in lossless mode
> > >
> > > diff -r 3eb2ec5922be -r 854f0f9aabf7 source/common/quant.cpp
> > > --- a/source/common/quant.cpp Wed Aug 05 12:20:01 2015 +0530
> > > +++ b/source/common/quant.cpp Tue Aug 11 14:30:27 2015 +0530
> > > @@ -228,9 +228,6 @@
> > >
> > >  void Quant::setQPforQuant(const CUData& ctu, int qp)
> > >  {
> > > -    m_tqBypass = !!ctu.m_tqBypass[0];
> > > -    if (m_tqBypass)
> > > -        return;
> >
> > that's a bit unexpected. what field needs to be initialized for
> > cu-lossless to work right?
> >
> > >      m_nr = m_frameNr ? &m_frameNr[ctu.m_encData->m_frameEncoderID] :
> > NULL;
> > >      m_qpParam[TEXT_LUMA].setQpParam(qp + QP_BD_OFFSET);
> > >      setChromaQP(qp + ctu.m_slice->m_pps->chromaQpOffset[0],
> > TEXT_CHROMA_U, ctu.m_chromaFormat);
> 
> When cu-lossless mode is set, we are comparing rate distortion cost for
> normal encoding and lossless encoding.
> For that tq_bypss flag is OFF for normal encoding and ON for lossless
> encoding. But since the qgSize is set to 32,
> qp value will change for depth 1 (32x32 CU). So for normal encoding we need
> to set the new qp value. The reason for hash mismatch
> was we were encoding the CU with the old qp value and signalling the
> decoder new qp value.

Ahh, Analysis::tryLossless() is setting m_quant.m_tqBypass directly. Ok,
this makes sense then.

Any idea why Search::checkIntra() does this:

  m_quant.m_tqBypass = !!cu.m_tqBypass[0];

it doesn't seem helpful

-- 
Steve Borho