Exploratory Examination of Correlation Matrices

A common first step of many data analyses that involve more than a very few variables is to run a correlation matrix of all variables and then examine it for expected (and unexpected) significant relations.

When this is done, you need to be aware of the general nature of statistical significance. Specifically, if you run many tests (in this case, many correlations), then significant results will be found surprisingly often due to pure chance.

For example, by definition, a coefficient significant at the .05 level will occur by chance once in every 20 coefficients. There is no automatic way to weed out the true correlations. Thus, you should treat all results that were not predicted or planned with particular caution and look for their consistency with other results. Ultimately, though, the most conclusive (although costly) control for such a randomness factor is to replicate the study.

This issue is general and it pertains to all analyses that involve multiple comparisons and statistical significance. This problem is also briefly discussed in the context of post-hoc comparisons of means and the Breakdowns option.