In their recent paper, Forbes et al. (2019; FWMK) evaluate the replicability of network models in two studies. They identify considerable replicability issues, concluding that "current 'state-of... Show moreIn their recent paper, Forbes et al. (2019; FWMK) evaluate the replicability of network models in two studies. They identify considerable replicability issues, concluding that "current 'state-of-the-art' methods in the psychopathology network literature [ horizontal ellipsis ] are not well-suited to analyzing the structure of the relationships between individual symptoms". Such strong claims require strong evidence, which the authors do not provide. FWMK identify low replicability by analyzing point estimates of networks; contrast low replicability with results of two statistical tests that indicate higher replicability, and conclude that these tests are problematic. We make four points. First, statistical tests are superior to the visual inspection of point estimates, because tests take into account sampling variability. Second, FWMK misinterpret the statistical tests in several important ways. Third, FWMK did not follow established recommendations when estimating networks in their first study, underestimating replicability. Fourth, FWMK draw conclusions about methodology, which does not follow from investigations of data, and requires investigations of methodology. Overall, we show that the "poor replicability "observed by FWMK occurs due to sampling variability and use of suboptimal methods. We conclude by discussing important recent simulation work that guides researchers to use models appropriate for their data, such as nonregularized estimation routines. Show less
Distance association models constitute a useful tool for the analysis and graphical representation of cross-classified data in which distances between points inversely describe the association... Show moreDistance association models constitute a useful tool for the analysis and graphical representation of cross-classified data in which distances between points inversely describe the association between two categorical variables. When the number of cells is large and the data counts result in sparse tables, the combination of clustering and representation reduces the number of parameters to be estimated and facilitates interpretation. In this article, a latent block distance-association model is proposed to apply block clustering to the outcomes of two categorical variables while the cluster centers are represented in a low dimensional space in terms of a distance-association model. This model is particularly useful for contingency tables in which both the rows and the columns are characterized as profiles of sets of response variables. The parameters are estimated under a Poisson sampling scheme using a generalized EM algorithm. The performance of the model is tested in a Monte Carlo experiment, and an empirical data set is analyzed to illustrate the model. Show less
Careless responding by mental health patients on self-report assessments is rarely investigated in routine care despite the potential for serious consequences such as faulty clinical decisions. We... Show moreCareless responding by mental health patients on self-report assessments is rarely investigated in routine care despite the potential for serious consequences such as faulty clinical decisions. We investigated validity indices most appropriate for detecting careless responding in routine outcome monitoring (ROM) in mental health-care. First, we reviewed indices proposed in previous research for their suitability in ROM. Next, we evaluated six selected indices using data of the Brief Symptom Inventory and the Mood and Anxiety Symptom Questionnaire from 3,483 outpatients. Simulations showed that for typical ROM scales the Lmax Lmax index, Mahalanobis distance, and inter-item standard deviation may be too strongly confounded with the latent trait value to compare careless responding across patients with different symptom severity. Application of two different classification methods to the validity indices did not converge in similar prevalence estimates of careless responding. Finally, results suggest that careless responding does not have a substantial biasing effect on scale-score statistics. We recommend the lpz lzp person-fit index to screen for random careless responding in large ROM data sets. However, additional research should further investigate methods for detecting repetitive responding in typical ROM data and assess whether there are specific circumstances in which simpler validity statistics or direct screening methods perform similarly as the lpz lzp index. Show less
Careless responding by mental health patients on self-report assessments is rarely investigated in routine care despite the potential for serious consequences such as faulty clinical decisions. We... Show moreCareless responding by mental health patients on self-report assessments is rarely investigated in routine care despite the potential for serious consequences such as faulty clinical decisions. We investigated validity indices most appropriate for detecting careless responding in routine outcome monitoring (ROM) in mental health-care. First, we reviewed indices proposed in previous research for their suitability in ROM. Next, we evaluated six selected indices using data of the Brief Symptom Inventory and the Mood and Anxiety Symptom Questionnaire from 3,483 outpatients. Simulations showed that for typical ROM scales the index, Mahalanobis distance, and inter-item standard deviation may be too strongly confounded with the latent trait value to compare careless responding across patients with different symptom severity. Application of two different classification methods to the validity indices did not converge in similar prevalence estimates of careless responding. Finally, results suggest that careless responding does not have a substantial biasing effect on scale-score statistics. We recommend the person-fit index to screen for random careless responding in large ROM data sets. However, additional research should further investigate methods for detecting repetitive responding in typical ROM data and assess whether there are specific circumstances in which simpler validity statistics or direct screening methods perform similarly as the index. Show less
Structural equation modeling is a common technique to assess change in longitudinal designs. However, these models can become of unmanageable size with many measurement occasions. One solution is... Show moreStructural equation modeling is a common technique to assess change in longitudinal designs. However, these models can become of unmanageable size with many measurement occasions. One solution is the imposition of Kronecker product restrictions to model the multivariate longitudinal structure of the data. The resulting longitudinal three-mode models (L3MMs) are very parsimonious and have attractive interpretation. This paper provides an instructive description of L3MMs. The models are applied to health-related quality of life (HRQL) data obtained from 682 patients with painful bone metastasis, with eight measurements at 13 occasions; before and every week after treatment with radiotherapy. We explain (1) how the imposition of Kronecker product restrictions can be used to model the multivariate longitudinal structure of the data, (2) how to interpret the Kronecker product restrictions and the resulting L3MM parameters, and (3) how to test substantive hypotheses in L3MMs. In addition, we discuss the challenges for the evaluation of (differences in) fit of these complex and parsimonious models. The L3MM restrictions lead to parsimonious models and provide insight in the change patterns of relationships between variables in addition to the general patterns of change. The L3MM thus provides a convenient model for multivariate longitudinal data, as it not only facilitates the analysis of complex longitudinal data but also the substantive interpretation of the dynamics of change. Show less
Whenever multiple regression is applied to a multiply imputed data set, several methods for combining significance tests for R2 and the change in R2 across imputed data sets may be used: the... Show moreWhenever multiple regression is applied to a multiply imputed data set, several methods for combining significance tests for R2 and the change in R2 across imputed data sets may be used: the combination rules by Rubin, the Fisher z-test for R2 by Harel, and F-tests for the change in R2 by Chaurasia and Harel. For pooling R2 itself, Harel proposed a method based on a Fisher z transformation. In the current article, it is argued that the pooled R2 based on the Fisher z transformation, the Fisher z-test for R2, and the F-test for the change in R2 have some theoretical flaws. An argument is made for using Rubin’s method for pooling significance tests for R2 instead, and alternative procedures for pooling R2 are proposed: simple averaging and a pooled R2 constructed from the pooled significance test by Rubin. Simulations show that the Fisher z-test and Chaurasia and Harel’s F-tests generally give inflated type-I error rates, whereas the type-I error rates of Rubin’s method are correct. Of the methods for pooling the point estimates of R2 no method clearly performs best, but it is argued that the average of R2’s across imputed data set is preferred. Show less
Steinley, Hoffman, Brusco, and Sher (2017) proposed a new method for evaluating the performance of psychological network models: fixed-margin sampling. The authors investigated LASSO regularized... Show moreSteinley, Hoffman, Brusco, and Sher (2017) proposed a new method for evaluating the performance of psychological network models: fixed-margin sampling. The authors investigated LASSO regularized Ising models (eLasso) by generating random datasets with the same margins as the original binary dataset, and concluded that many estimated eLasso parameters are not distinguishable from those that would be expected if the data were generated by chance. We argue that fixed-margin sampling cannot be used for this purpose, as it generates data under a particular null-hypothesis: a unidimensional factor model with interchangeable indicators (i.e., the Rasch model). We show this by discussing relevant psychometric literature and by performing simulation studies. Results indicate that while eLasso correctly estimated network models and estimated almost no edges due to chance, fixed-margin sampling performed poorly in classifying true effects as “interesting” (Steinley et al. 2017, p. 1004). Further simulation studies indicate that fixed-margin sampling offers a powerful method for highlighting local misfit from the Rasch model, but performs only moderately in identifying global departures from the Rasch model. We conclude that fixed-margin sampling is not up to the task of assessing if results from estimated Ising models or other multivariate psychometric models are due to chance. Show less
