Example 10: Confidence Intervals and Special Tests on the Multiple Correlation
The exact distribution of the squared multiple correlation coefficient is difficult to compute. Consequently, a number of power analysis books and programs rely on approximations that, while generally reasonably accurate, can result in non-negligible errors under some conditions (Gatsonis & Sampson, 1989). STATISTICA Power Analysis calculates the exact distribution of the squared multiple correlation under the assumption that the observed variables have a multivariate normal distribution.
In this example, we examine a number of power analysis, sample size computation, interval estimation, and hypothesis-testing procedures that can be performed with the STATISTICA Power Analysis module.
- Confidence Intervals on the Coefficient of Determination
- The coefficient of determination, R2, is reported routinely as a byproduct of a multiple regression analysis. Along with the sample R2 value, a significance level for the test of the 1-tailed hypothesis
against the alternative
H1: P2 > 0
is usually provided. (P2 is the population equivalent of R2.) These values frequently are not very informative, because (a) the estimate of the coefficient of determination is often subject to substantial sampling error, and (b) the "nil" hypothesis that the coefficient of determination is zero is often completely unrealistic.
A confidence interval on P2 is a much more effective way to present the information that has been obtained from the sample than simply quoting a point estimate and a significance level. For one thing, the standard hypothesis test can be performed with the confidence interval, because the hypothesis test will reject at the α significance level if and only if the 1 - 2α confidence interval fails to include zero. Moreover, the width of the confidence interval provides a valuable indication of the precision of estimation obtained from the analysis.
To perform the confidence interval estimation, select Power Analysis from the Statistics menu to display the Power Analysis and Interval Estimation Startup Panel. From the Startup Panel, select Interval Estimation and Squared Multiple Correlation.
Click the OK button to display the Multiple R2: Interval Estimation dialog.
Now suppose that you had just completed a multiple regression analysis in which there were 15 predictor variables, and a sample size of N = 104. The observed sample value for R2 is reported as .30.
To analyze these values, enter 0.30 in the Observed R2 box, 104 in the Sample Size (N) box, 15 in the number of Ind. Vars (k) box, 0.05 in the Alpha box, and 0.90 in the Conf. Level box. Make sure that the 1-tailed (P2 = 0) option button is selected under Type of Hypothesis. The dialog should look like this:
Now click the Compute button. STATISTICA displays a spreadsheet with the results of calculations.
First, note that STATISTICA reports a probability level for the observed R2 of .0039. The observed value of 0.30 is significant beyond the .01 level. The confidence interval for P2, however, demonstrates that the population coefficient of determination has not been measured with a high degree of precision. The 90% confidence interval ranges from .0595 to .3257. The program also reports a 90% lower bound, or 1-tailed confidence interval. This suggests that you can be 90% confident that the population coefficient of determination is no less than .085. This demonstrates that although the observed R2 is statistically significant, there is not strong evidence that the proportion of variance accounted for by the predictor variables is substantial.
STATISTICA also reports post hoc confidence intervals on statistical power. In this case, the confidence interval on power is quite wide, ranging from .2539 to .9952. This is not surprising because power is directly related to the size of the population squared multiple correlation, and the confidence interval on that quantity is quite wide. What this means is that the result we just observed is consistent with the notion that power was quite low, or quite high.
The confidence interval on P2 can also be converted into a confidence interval on the sample size needed to achieve the current power goal. This interval says that we can be 90% confident that the sample size needed to achieve a power of .90 in this study is between 68 and 390. So the sample size actually employed (104) may be about 50% too large, or may be, in the long run, way too small.
These confidence intervals dramatize that statistical significance need not translate into usable levels of precision of estimation.
- Testing a Hypothesis of Minimal Correlation
- In many applications, the hypothesis of zero multiple correlation is not a particularly interesting hypothesis to test, because it is almost certainly false. A more interesting hypothesis to test might be that the coefficient of determination is less than or equal to a target value (representing some minimal acceptable level of variance accounted for by the regression). For example, Murphy and Myors (1998) suggest testing minimal effect values like .01 or .05. In this section, we demonstrate how to
- Find critical values for testing any hypothesis about the squared multiple correlation;
- Find a probability level for an observed value of R2;
- Assess power for testing such a hypothesis against a specified alternative that P2 = .30;
- Calculate required sample size for a particular power goal, when testing a hypothesis of "minimal effect."
Suppose, for example, we wanted to test the null hypothesis that P2 ≤ .05 against the alternative that P2 > .05, in a situation where there are 10 predictor variables, the sample size is 200, and the significance level is .01. To obtain the critical value of R2 for the test, simply use the probability distribution calculator.
Go back to the Startup Panel (by pressing the ESC key) and select Probability Distributions and Squared Multiple Correlation.
Click the OK button to display the Multiple R2 Probability Calculator dialog.
This is a 1-tailed test, so our critical value will have a cumulative probability of .99 in order for α to be .01. Enter .05 in the P2 box, 10 in the Ind. Vars. (k) box, 200 in the Sample Size (N) box, and 0.99 in the Cum. p box. Select the R2 option button under Compute to specify the Observed R2 as the quantity to compute. Finally, click the Compute button to obtain the critical value. The dialog box should look like this:
We have determined the critical value to be .1915626.
Determining the cumulative probability, or the "p-value," is easy in this dialog. To calculate the 1-tailed significance level, ensure that (a) the (1 - Cumulative p) check box is selected, and that the 1 - p option button is selected under Compute. Enter the Observed R2 for which you want to compute the significance level. For example, suppose you observe a sample R2 value of .205, and want to know the significance level. Enter .205 in the Observed R2 box, and click the Compute button. The dialog should appear as below:
To compute power in this situation when the true coefficient of determination is .30, display the Multiple R2 Power Calculation Parameters dialog. Click the Back button to return to the Startup Panel, and select Power Calculation and Squared Multiple Correlation.
Click the OK button to display the Multiple R2 Power Calculation Parameters dialog. Enter the actual value of P2 as .30, 0.05 as the C2(Null P2), 10 as the Ind. Vars. (k), 200 as the N, and 0.01 as the Alpha. Also, be sure to select the appropriate 1-tailed hypothesis by selecting the 1-tailed (P2 <= C2) option button under Type of Hypothesis. The dialog should look like this when you are ready to compute:
Click the OK button to display the Multiple R2 Power Calculation Results dialog.
To calculate power of this test of minimal effect, click the Calculate Power button. This will display a spreadsheet with the results of the power calculation.
In this case, we see that the power is quite high, perhaps higher than we need. Money and time might be saved by reducing the sample size. This raises the question, "How big a sample size would we need to have power of .90 in this situation?"
To perform this calculation, go back to the Startup Panel (by pressing the ESC key twice), and select Sample Size Calculation and Squared Multiple Correlation.
Click the OK button to display the Multiple R2 Sample Size Parameters dialog.
As you can see, the parameters are already correct for performing the desired calculation. Click the OK button to display the Multiple R2 Sample Size Calculation Results dialog.
Click the Calculate N button to calculate the required sample size.
It turns out that a sample size of 116 would be adequate to produce power of .90 in this situation.
See also, Power Analysis - Index.
