How do you use the leading coefficient test to determine the end behavior of the polynomial function ##f(x)= -5(x2+1)(x-2)##?

If the leading coefficient is negative, the polynomial function will eventually decrease to negative infinity; if the leading coefficient is positive, the polynomial function will eventually increase to positive infinity.

This answer assumes by leading coefficient you mean the coefficient of the highest powered ##x## term (the normal usage).

If ##g(x)## is a polynomial with greatest degree ##n## then if ##m > n##, the absolute value of ##x^m## will be greater than the absolute value of ##g(x)## once ##x## becomes sufficiently large.

For example if ##g(x) = 5x^2 - 4x +12## ##x^3## will have an absolute value greater than ##g(x)## provided x is greater than 3.

Since the term with the highest exponent will eventually provide a value that exceeds the combined value of all other terms, the sign of that term determines the eventual direction of the function value.