[x265] ASM crash in r6706

[Robert Bacs]

On 4/15/2014 9:16 PM, Steve Borho wrote:
> Is your source video publicly available? 
My source was a y4m created with ffmpeg, but you can use this one 
http://media.xiph.org/video/derf/y4m/bowing_cif.y4m, I just tried and 
got a similar call-stack:

#0  0x0051995e in x265_quant_sse4 ()
#1  0x00603ba7 in x265::TComTrQuant::getTUEntropyCodingParameters (cu=
     0x4789020, result=..., absPartIdx=74824640, log2TrSize=16,
     ttype=x265::TEXT_CHROMA)
     at E:/temp/x265_clean/source/Lib/TLibCommon/TComTrQuant.h:171
#2  0x005b5da8 in x265::TComTrQuant::transformNxN (this=0x4360528,
     cu=0x4789020, residual=0x47e5d20, stride=32, coeff=0x475bbc0, 
trSize=16,
     ttype=x265::TEXT_CHROMA, absPartIdx=0, lastPos=0x27af748,
     useTransformSkip=false, curUseRDOQ=true)
     at e:/temp/x265_clean/source/Lib/TLibCommon/TComTrQuant.cpp:349
#3  0x0059c289 in x265::TEncSearch::xIntraCodingChromaBlk (this=0x4360240,
     cu=0x4789020, trDepth=1, absPartIdx=0, absPartIdxStep=64, fencYuv=
     0x47e9fc0, predYuv=0x47d8ce0, resiYuv=0x47e3c88, outDist=@0x27afab4: 0,
     chromaId=1, bReusePred=false)
     at e:/temp/x265_clean/source/Lib/TLibEncoder/TEncSearch.cpp:579
#4  0x0059f6a2 in x265::TEncSearch::xRecurIntraChromaCodingQT (
     this=0x4360240, cu=0x4789020, trDepth=1, absPartIdx=0, 
fencYuv=0x47e9fc0,
     predYuv=0x47d8ce0, resiYuv=0x47e3c88, outDist=@0x27afab4: 0)
     at e:/temp/x265_clean/source/Lib/TLibEncoder/TEncSearch.cpp:1304
#5  0x0059f7a6 in x265::TEncSearch::xRecurIntraChromaCodingQT (
     this=0x4360240, cu=0x4789020, trDepth=0, absPartIdx=0, 
fencYuv=0x47e9fc0,
     predYuv=0x47d8ce0, resiYuv=0x47e3c88, outDist=@0x27afab4: 0)
     at e:/temp/x265_clean/source/Lib/TLibEncoder/TEncSearch.cpp:1323
#6  0x005a2222 in x265::TEncSearch::estIntraPredChromaQT (this=0x4360240,
     cu=0x4789020, fencYuv=0x47e9fc0, predYuv=0x47d8ce0, resiYuv=0x47e3c88,
     reconYuv=0x47e6d90)
     at e:/temp/x265_clean/source/Lib/TLibEncoder/TEncSearch.cpp:1917
#7  0x005b4121 in x265::TEncCu::xCheckRDCostIntra (this=0x43604ac,
     outBestCU=@0x477e568: 0x477e7e8, outTempCU=@0x477e590: 0x4789020,
     partSize=x265::SIZE_2Nx2N)
     at e:/temp/x265_clean/source/Lib/TLibEncoder/TEncCu.cpp:1397
#8  0x005b095b in x265::TEncCu::xCompressIntraCU (this=0x43604ac,
     outBestCU=@0x477e568: 0x477e7e8, outTempCU=@0x477e590: 0x4789020, 
depth=0)
     at e:/temp/x265_clean/source/Lib/TLibEncoder/TEncCu.cpp:598
#9  0x005b03e0 in x265::TEncCu::compressCU (this=0x43604ac, cu=0x5cb9188)
     at e:/temp/x265_clean/source/Lib/TLibEncoder/TEncCu.cpp:384
#10 0x005bdd30 in x265::CTURow::processCU (this=0x435fd98, cu=0x5cb9188,
     slice=0x5cbcdf0, bufferSbac=0x0, bSaveSBac=false)