Significance tests and estimates for R2 for multiple regression in multiply imputed datasets: A cautionary note on earlier findings, and alternative solutions

Abstract

Whenever multiple regression is applied to a multiply imputed data set, several methods for

combining significance tests for R2 and the change in R2 across imputed data sets may be

used: the combination rules by Rubin, the Fisher z-test for R2 by Harel, and F-tests for the

change in R2 by Chaurasia and Harel. For pooling R2 itself, Harel proposed a method based on a Fisher z transformation. In the current article, it is argued that the pooled R2 based on the Fisher z transformation, the Fisher z-test for R2, and the F-test for the change in R2 have some theoretical flaws. An argument is made for using Rubin’s method for pooling significance tests for R2 instead, and alternative procedures for pooling R2 are proposed: simple averaging and a pooled R2 constructed from the pooled significance test by Rubin. Simulations show that the Fisher z-test and Chaurasia and Harel’s F-tests generally give inflated type-I error rates, whereas the type-I error rates of Rubin’s method are correct. Of the methods for pooling the point estimates of R2 no method clearly performs best, but it is argued that the average of R2’s across imputed data set is preferred.