How do you find the integral of ##tan^2(x) * sec^3(x) dx##?

See the explanation section, below.

Rewrite the integrand using ##tan^2x = sec^2x-1##.

Let's give the integral we want the name ##I##

##I = int tan^2xsec^3x dx = int (sec^5x-sec^3x)dx##

Next we'll integrate ##sec^5x## by parts.

##int sec^5x dx = int sec^3 x sec^2x dx##

Let ##u = sec^3 x## and ##dv = sec^2x dx##.

Then ##du = 3tanx sec^3x dx## and ##v = tanx##

We get

##int sec^5 x dx = sec^3x tanx - 3int tan^2x sec^3x dx##

Again, use ##tan^2x = sec^2 x-1## to get

##int sec^5 x dx = sec^3x tanx - 3int (sec^2 x-1) sec^3x dx##

##int sec^5 x dx = sec^3x tanx - 3int sec^5 dx + 3int sec^3x dx##

Which gets us

##4int sec^5 x dx = sec^3x tanx + 3int sec^3x dx##

and

##int sec^5 x dx = 1/4sec^3x tanx + 3/4 int sec^3x dx##

Recalling that the other integral we need is ##int sec^3x dx##, let's simplify our lives by writing:

##I = intsec^5xdx-intsec^3xdx##

## = underbrace(1/4sec^3x tanx + 3/4 int sec^3x dx)_(intsec^5 x dx)-intsec^3xdx##

## = 1/4sec^3x tanx -1/4 int sec^3x dx##

So now find ##int sec^3x dx = int secx sec^2x dx## using the same general approach and . You should get

##int sec^3x dx = 1/2secxtanx - 1/2int secx dx##

So at this point we have

##I = 1/4sec^3x tanx -1/4underbrace( [1/2secxtanx - 1/2int secx dx])_(intsec^3 x dx)##

## = 1/4sec^3x tanx -1/8secxtanx + 1/8 int secx dx##

Now ##int secx dx## can be evaluated in a couple of ways, the usual trick is to multiply by ##(secx+tanx)/(secx+tanx)## to get ##int 1/u du = ln absu = ln abs(secx+tanx)##

So finally we finish with

##I = 1/4sec^3x tanx - 1/8secxtanx + 1/8 ln abs(sec x + tan x ) +C##

Alternative form for ##int secx dx##

##intsecx dx## can also be found by substitution and partial fractions to get ##1/2ln abs((sinx+1)/(sinx-1))+C## (Yes, the is equivalent to ## ln abs(secx+tanx)+C##)