Hi John:
Thanks for your note. We appreciate and take seriously comments from thoughtful people such as yourself and have from the beginning been committed to continually improve the quality and utility of the NSSE project. It appears that you have read the NSSE Conceptual framework on our website, but we do report other information regarding NSSE psychometrics.
We view student engagement not as a single construct but rather as a domain of constructs that reflect the time and energy students put into their learning activities in college. When we developed the NSSE benchmarks in 1999, we consulted the higher education literature and expert opinion to group the items into the five content-related themes that could be used to convey broad measures of engagement and to focus conversations about institutional quality on the part of NSSE users and the public. At that point we calculated coefficient alphas for all NSSE benchmarks and other measurement scales developed later using more rigid psychometric methods. You can find NSSE reliability information here.
In addition to the benchmarks NSSE has a deep learning scale with three sub-scales: higher order thinking, integrative learning, and reflective learning. The deep learning scale was developed using factor analysis and our research has shown that they have pretty good psychometric properties. More and more studies show that deep learning can be a predictor of academic outcomes. You can find some of these studies in our papers and conference presentations section of our web site.
Gary Pike (2006) also summed smaller sets of NSSE questions to create 15 student-engagement scales (he labels "scalelets") often subsets of the NSSE benchmarks. Pike found that these provided "dependable metrics for assessing student engagement at the university, college, and department levels" and have "greater explanatory power and provide richer detail than the NSSE benchmarks."
The linkage between the student engagement concept and college outcomes has been confirmed many times in the literature by Pascarella and Terenzini (2005). The relationships between NSSE measures and outcomes is one of continuing study. For example, NSSE is currently participating in a study that will link liberal arts education, student engagement, and college outcomes like critical thinking ability. You may want to take a look at "Connecting the Dots: Multi-faceted Analyses of the Relationships between Student Engagement Results from the NSSE, and the Institutional Practices and Conditions that Foster Student Success," in which we connected the NSSE Benchmarks and student outcome measures such as grades and persistence.
We also ask students to self-report their gains in general education, practical competence, and personal and social development. The connections between NSSE benchmarks and student self-reported gains can be found here.
Please don't hesitate to contact me if you have other questions.
Best wishes.
George
Dear George,
We are in the beginning stages of our next accreditation review. In a number of conversations with a friend we exchanged some ideas about various assessment techniques/instruments. I expressed some concerns about the NSSE and my friend suggested I contact you for your response. I hope that's ok.
My concerns can be summarized as follows:
The construction of the Benchmarks did not follow from the factor analyses in the development samples, but seem to have been constructed on some other basis (with some reference to the results of the factor analyses, it seems). This doesn"t bother me, if the resulting scales are useful, but it makes me wonder why the factor analyses were done in the first place.
Cronbach's Alphas are not reported for the Benchmarks, nor are any other internal consistency indices, as far as I can see.
No validity data is reported for the Benchmarks, with respect to the relation of the Benchmarks to other similar instruments (save for statements about it), nor for the relations of the Benchmarks to measures of "deep learning".
The only data of this sort at all is the correlations with GPA, which are much too low to be of use. Certainly an argument is that grades are overdetermined and, thus, would be expected to have relatively low correlations with Benchmarks. However, if 1) the Benchmarks have high internal consistency, and 2) GPA at the senior year has much less error than grades at the first year (due to attenuation of random error over multiple measurements) , the correlations between the Benchmarks and senior GPA should be substantially higher than first-year GPA and should be at least high enough to warrant further use of the Benchmarks. All the data I"ve seen, including our own, indicate that senior and first-year correlations are similar and quite weak. (As a note, I constructed new scales using the original factor analyses results and correlated those scale scores with GPA for our students. The resulting values were lower, in most cases, than the Benchmark/GPA correlations.)
My general sense is that while the NSSE Benchmarks have intuitive appeal and very good "face validity", there is no empirical evidence that the Benchmarks do what they purport to do (i.e., measure activities that are associated with [not causal?] deep learning – my paraphrase).
Until and unless there is real evidence/data to support the claims that the NSSE Benchmarks measure something associated with (and causal by strong inference) deep learning (or its synonyms), I cannot recommend its use.
If I've missed the boat entirely and/or is there's new evidence to support the use of the NSSE, I shall gladly eat my words.
Respectfully,
John Doe
