Tony Gardner-Medwin
Posts: 14
 |
| 30/05/2007 07:39 |
|
Nigel, thanks for the stimulating comments in your review. It's interesting that the concerns you raise about CBM are all about use in summative assessment. Our main objective has been to improve students' habits of thought and study while learning (formative assessment), and it was surprising to me, when CBM was used in UCL exams, that the concerns you express about summative assessment simply didn't materialise.
You ask about standard setting. As we show in the paper, CBM can generate a score (CBS) that is on average equivalent (for both T/F and best-of-5 MCQs) to the % correct above chance. I think you may have misunderstood this "% correct above chance" concept - known perhaps more simply at Imperial as "% knowledge". This has nothing to do with confidence ratings, but is simply a way of expressing the conventional score ("number correct"), scaled so that complete guesses would give 0% and perfect scores 100%. For example, on a T/F exam guesses would yield on average 50% correct, so a passmark of 75% correct answers corresponds to "50% knowledge" (half way between guesses and perfection). Similarly with best-of-5 exam Qs, guesses would on average give 20% correct, so it is 60% correct that corresponds to "50% knowledge". Whatever passmark you set based on criterion referenced standard setting procedures, if you scale this to "% knowledge" then the equivalent CBS passmark will be the same, as shown in the empirical relationship in Fig. 2.
Yes, a weak but confident student may occasionally do well through luck, as with any assessment scheme. The nature of CBM means however that both in theory and in practice a weak student will lose by claiming confidence where it is not justified. A strong student will also lose through inappropriate diffidence (too often choosing low certainty levels). But experience shows that practised students become well calibrated to use certainty levels appropriately (Gardner-Medwin & Gahan, 2003: http://www.ucl.ac.uk/~ucgbarg/tea/caa03.doc ). The students who do best under CBM, with the same number of correct answers, are those with better than average ability to identify which of their answers are reliable and which are based on uncertain knowledge or reasoning. Knowledge, after all, is not just a matter of being right but of knowing that you are right.
The query about whether CBM enhances stress is interesting, and I hadn't encountered it before. It doesn't seem likely to be a major student concern, since in a 2005 survey, less than 1/3 of UCL students with experience of CBM in year 1, 2 exams voted to drop CBM from exams. Imperial College cautiously allowed extra time when it first introduced CBM into formative exams, but it became clear that extra time was not needed. It is an interesting fact about the brain that when it comes up with an answer, this seems to come packaged with a certainty judgement. The judgement may be wrong, which can in many walks of life be a disaster. But it can be refined or corrected by reflection. CBM helps to train students to make such judgements carefully, correctly and realistically and rewards them accordingly.
Does CBM assess reasoning? Certainly CBM can be used with questions that require reasoning. CBM puts a premium on care in such reasoning. Many of our incoming students - I see this especially in maths for medical students - have got in the habit of giving answers with very little thought. Since they are highly selected students, these answers are often right. But at university, this simply isn't good enough. If it were possible, we would of course always like to assess and comment on the quality or steps in a student's reasoning, and not just base a mark on the final answer and confidence expressed for whether it is right. But this requires 'hand' marking, and is not always possible. CBM motivates extra care in reasoning, realistic self-appraisal and checking. There is surely no way that reflection about the quality of reasoning is in any way different or less valuable than reasoning itself.
Thanks again - Tony G-M
|
|
|
|
|
