Review of Gray and McKitrick REAP papers and Questions for Discussion
Stephen C. Ehrmann, Director, The Flashlight Program, The TLT Group (http://www.tltgroup.org/flashlightP.htm)
Both papers argue that involvement of the instructional staff is key to institutionalization of assessment.
At the Naval Academy, a process of institution-wide goal setting (1999) was followed by an effort to encourage instructors to incorporate enhanced in their courses and departments. The effort began with training for interested volunteers who were also seen as opinion leaders in their disciplines. Tactics included travel by volunteers to conferences and other institutions, local workshops, colloquia, poster sessions, and development of assessment plans for programs and departments. An office of Academic Assessment was created and its director worked with the Director of Teaching and Learning. This process resulted in changes in assessment if departments found that their initial plans were unworkable and also led, “in a number of cases, to course or curricular changes.” The process described by Gray is quite systematic, but the paper was too brief to understand whether it has had success in creating a cycle of success and dissemination among faculty, or in fostering improved student learning. .
The McKitrick paper provides more detail about both the goals being assessed and the methods in use. In contrast to the Naval Academy’s focus on departments, SUNY Binghamton worked on general education and, in this paper, specifically on critical thinking in general education courses. Program leaders engaged interested instructors in discussions of student progress toward institutional goals. They did so through a Delphi survey (two surveys, with the second being used to create consensus by feeding back results from the first) and meetings. Evidence was gathered through instructors, the library, internship supervisors and other sources. The discussion has led to plans for assessment of ‘learning in upper-division critical thinking and information management courses.’ There were no examples of specific changes in how critical thinking was being assessed. McKitrick does mention, however, that assessment of information literacy has led to insight into some specific problems and changes in teaching.
Questions for the authors
- Peter, I couldn’t tell whether the effort in the first half of the decade at the Naval Academy was mainly to assess courses and programs on their own terms and/or whether the effort was focused on assessment of progress toward the academic goals defined in 1999. Could you clarify that point? Does the effort seem to have resulted in any improvement of performance on these Academy goals so far?
- Sean, you wrote about the initial bumps and starts, “Initial meetings by assessment staff with faculty and staff members evidenced some confusion about what role assessment would play in the tenure and promotion process, evaluations of teaching, and control over resources at department and program levels.” Can you tell us more about this, and whether/how things were clarified?
- Sean, your paper describes an effort to engage a large fraction of the instructors of general education courses in thinking together about critical thinking and its assessment. The paper mentions 90% of faculty respondents to a survey liked the engagement effort (what fraction of faculty responded to the survey?) What do you see as the biggest success of this effort to engage faculty? The biggest frustration or disappointment? Based on your experience, what would you advise other institutions to do differently than Binghamton did?
- For both authors, suppose all external pressures for assessment were now removed after 2007. Would assessment continue to grow and develop at your institution? Have there been enough individual and programmatic success stories that the move toward a culture of goals and evidence would continue?
The remaining questions are for all participants in this discussion:
- How engaged is your instructional staff in the improvement of assessment? For example, what fraction of your instructors have attended more than one assessment training session or event over the years? What strategies seem most promising for engaging a larger fraction of your academic staff in improving assessment?
- What tools or techniques for assessing learning have been MOST easily and widely shared among instructors and departments? (I’m talking about tools for assessing the outcomes and the process of learning) Rubric designs? Survey or feedback forms? Particular types of performance tests such as the Force Concept Inventory in physics?
- How adequate by itself is assessment data, by itself, for showing where and how to improve learning? For example, I’m guessing that disappointing data about student writing skill at graduation would not, by itself, provide much guidance about how to improve student writing. From assessment and other sources, are your instructors getting the evidence they need in order to see HOW to improve learning for their students? (for some of my own thoughts on this issue, see the second reference below).
Related Resources from the Facilitator
“Creating a Culture of Evidence – Case Studies” http://www.tltgroup.org/Flashlight/Handbook/Instns_Data.htm
“What Outcomes Assessment Misses” (why outcomes data often provides inadequate guidance for how to outcomes, what to do about it)
http://www.tltgroup.org/programs/outcomes.html |
