What unit of force is used in the universal gravitational constant?

That depends on the unit system used, but in the SI (Systeme Internationale) system, units are as shown here:

The gravitational constant ##G=6.67xx10^-11## ##m^3 kg^-1 s^-2##. Alternatively, and equivalently, the units can be written as ##Nm^2kg^-2##, in which case the force unit is the newton (N).

The gravitational force expression is:

##F=(Gm_1m_2)/r^2##

If we rearrange to make ##G## the subject:

##G=(Fr^2)/(m_1m_2)##

We can replace the symbols with the appropriate units, as follows:

##F## ##(N)## ##r## ##(m)## ##m## ##(kg)##

Then the units for ##G## we get are ##Nm^2kg^-2##.

One newton is one ##kgms^-2##, which is how we translate between the two sets of units.