What is the difference between a correlation matrix and a covariance matrix?

A covariance matrix is a more generalized form of a simple correlation matrix.

Correlation is a scaled version of covariance; note that the two parameters always have the same sign (positive, negative, or 0). When the sign is positive, the variables are said to be positively correlated; when the sign is negative, the variables are said to be negatively correlated; and when the sign is 0, the variables are said to be uncorrelated.

Note also that correlation is dimensionless, since the numerator and denominator have the same physical units, namely the product of the units of ##X## and ##Y##.

Best Linear Predictor Suppose that ##X## is a random vector in ##RR^m## and that ##Y## is a random vector in ##RR^n##. We are interested in finding the function of ##X## of the form ##a+bX##, where ##a in RR^n## and ##b in RR^{nxxm}##, that is closest to ##Y## in the mean square sense. Functions of this form are analogous to linear functions in the single variable case.

However, unless ##a=0##, such functions are not in the sense of linear algebra, so the correct term is affine function of ##X##. This problem is of fundamental importance in statistics when random vector ##X##, the predictor vector is observable, but not random vector ##Y##, the response vector.

Our discussion here generalizes the one-dimensional case, when ##X## and ##Y## are random variables. That problem was solved in the section on Covariance and Correlation.

http://www.math.uah.edu/stat/expect/Covariance.html