<div dir="ltr"><span style="font-size:small;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">While running x264 under Clang's UndefinedBehaviorSanitizer, I found a few videos and settings that trigger an<span> </span></span><span class="gmail-il" style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">integer</span><span style="font-size:small;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span> </span></span><span class="gmail-il" style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">overflow</span><span style="font-size:small;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline"><span> </span>in<span> </span></span><span class="gmail-il" style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">mb_analyse_intra</span><span style="font-size:small;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">. The following line can occasionally try to take satd[i_pred_mode] below INT_MIN (presently encoder/</span><span class="gmail-il" style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">analyse</span><span style="font-size:small;text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">.c:907):</span><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">    satd[i_pred_mode] -= 3 * lambda;</div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">I've locally worked around this by using an int64_t and clamping the result back to an int32_t, however I'd imagine that's not desirable for 32-bit targets. That said, I've appended the patch. If this looks good, can someone merge it? If not, can someone help me fix this in a more appropriate way?</div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">Thanks,</div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial">-Jeremy</div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial"><br></div><div style="font-size:small;text-decoration-style:initial;text-decoration-color:initial"><div>diff --git a/encoder/<span class="gmail-il">analyse</span>.c b/encoder/<span class="gmail-il">analyse</span>.c</div><div>index fb241a7d..564fe73a 100644</div><div>--- a/encoder/<span class="gmail-il">analyse</span>.c</div><div>+++ b/encoder/<span class="gmail-il">analyse</span>.c</div><div>@@ -904,7 +904,8 @@ static void<span> </span><span class="gmail-il">mb_analyse_intra</span>( x264_t *h, x264_mb_analysis_t *a,<span> </span><span class="gmail-il">int</span><span> </span>i_satd_inter</div><div>                     ALIGNED_ARRAY_16( int32_t, satd,[9] );</div><div>                     h->pixf.intra_mbcmp_x3_4x4( p_src_by, p_dst_by, satd );</div><div>                     <span class="gmail-il">int</span><span> </span>favor_vertical = satd[I_PRED_4x4_H] > satd[I_PRED_4x4_V];</div><div>-                    satd[i_pred_mode] -= 3 * lambda;</div><div>+                    int64_t new_satd = (int64_t)satd[i_pred_mode] - 3 * (int64_t)lambda;</div><div>+                    satd[i_pred_mode] = X264_MAX(INT_MIN, new_satd);</div><div>                     i_best = satd[I_PRED_4x4_DC]; a->i_predict4x4[idx] = I_PRED_4x4_DC;</div><div>                     COPY2_IF_LT( i_best, satd[I_PRED_4x4_H], a->i_predict4x4[idx], I_PRED_4x4_H );</div><div>                     COPY2_IF_LT( i_best, satd[I_PRED_4x4_V], a->i_predict4x4[idx], I_PRED_4x4_V );</div></div><div class="gmail-yj6qo" style="font-size:small;text-decoration-style:initial;text-decoration-color:initial"></div><br class="gmail-Apple-interchange-newline"><br></div>