[vlc-devel] [PATCH] check for crazy values in rtsp npt range

[Tobias Güntner]

Hello!

> +                if(d_length > INT64_MAX)
> +                    *pi64 = INT64_MAX;
> +                else
> +                    *pi64 = (int64_t)d_length;

This will fail if d_length == pow(2, 63).

When comparing a double and an integer, the integer (no matter what
size) is first converted to a double. But INT64_MAX (usually 2^63-1) has
no exact double representation; the nearest
representable value is INT64_MAX+1 (2^63). Therefore, (pow(2, 63) >
INT64_MAX) is actually false!

I'm not sure how to work around this. feclearexcept/llrint/fetestexcept
perhaps?

Regards,
Tobias