The first requirement for the system I’ve been putting together as a thought experiment that would accredit memorisation (see my three previous posts for some background) would be an infinite set of well tagged questions.
I think this is the easiest part of the system to put in place. We are all aware of the success of crowdsourcing as a way to provide content (think wikipedia). So why don’t we put together an open source question base?
Since this learning system is simply about fluency of recall, all we need are questions about stuff. And lots of them.
It isn’t simply about the questions. in order to make this a memorisation/learning environment, the questions have to be tagged – well tagged. This is necessary so that users can focus on their own learning desires.
The kind of tagging that would make this system useful has three varieties of tags: content domain, source, and event.
The content domain tags are the most obvious. Libraries have spent centuries (literally) organising knowledge into content domains. There are wonderful hierarchal systems that allow users to find learning resources (books, articles, papers, websites, posts, pictures, videos – and who knows what else) within a specific content domain. We haven’t been all that great at tagging these resources, but there’s no reason we can’t start. Within the new question base, an easy to use content domain tagging system is a must.
The second set of tags ahas to do with sources. Knowledge is found somewhere, and if questions can be tagged with a specific source, that makes them all the more powerful. Specific books, journal articles, or web-articles (think wikipedia) would allow users (both learners and contributors) to specify exactly where the information comes from that needs to be memorised to a fluent level. Teachers (face to face or virtual) could then specify both content domain and source, along with the required level of proficiency, for an event (discussion, seminar series etc.) required for the learner to be able to participate fluently.
Finally, event tags could be included so that learners could prepare themselves for the kind of events specified above. They could even be specified for traditional assessment events (mid-term or final exams).
Properly tagged, an infinite number of questions embedded in a threshold learning system, could provide learners and educators with an invaluable tool for the foundational learning we call memorisation.
Information abundance means that learners have unprecedented access to information.This coupled with what we know about student engagement in academic study means that we might want to rethink the way we approach assessments. I believe that social media (SM) tools provide us with unique opportunities to asses in ways that weren’t even possible a few years ago. Using SM tools can provide opportunities not available using traditional assessment tools.
Regular readers will know that I am a huge fan of blogging as a form of assessment using SM. Some of the advantages (from my perspective) for the use of SM blogging include public exposure, and the ability to comment. Other advantages, that I think are relatively minor (from my perspective as a teacher) but are important to students, are: solid platforms, 24/7 access, universal availability, and simple and quasi-familiar interfaces. I think that the advantages that the students focus on are important to their ability to do the work, however, the advantages I focus on are real advantages for learning.
Although a few students (and a great many teachers) fear public exposure when it comes to assessment, I believe that it is of great benefit in the learning process. One of the hallmarks of authentic assessment is that the assessment is a closer reflection of the type of activity that is expected in real life. Seldom is serious writing done for the purposes of having a single individual read it, and then have it disappear. That is how most traditional writing assessments are done in HE today, with few exceptions. When students work is put up for public display, several things happen, they take more care in their work, they begin to produce work that will impress their peers, friends and family (you wouldn’t believe how many of my students invite their parents to participate in their learning this way), they look at each other’s work as models of good practise (how often does that happen with traditional assessments), they monitor each other’s work for unfair practise (with serious repercussions), and they are available for the wider community to engage with them.
My students tell me that after a few weeks of producing their weekly blog posts, I begin to disappear from their thinking when they are writing. They begin to write for their audience, in which I become a minor player. They write to convince those who will be making comments on their work, not for me, who will be grading the work. They present coherent arguments, backed by evidence and clear thinking, that allow them to get across the points that they want to make.
They also tend to invite others, not involved in the class, to read their work. They will interlink their various SM tools so that when they publish a post, it goes out to their friends and family on Facebook, Google+, and Twitter. When I see them in class and ask about a mother’s (or friends, or cousin’s) comment on their post, they invariably blush and say, “That’s my mother (or cousin or whatever) – I don’t know how she got on there.” Well, I do! The students have invited them in. In many cases, this is just an extension of them bringing the pictures home that they drew in grade 2, and looking for some measure of praise. This is great. Anyone who is going to write something that their mother is going to read is going to make sure she has a reason to say ‘well done!’. Why wouldn’t we, as educators, want to take advantage of this?
They have to read each other’s work in order to write (required) comments every week. As a part of the model I use, I write a short paragraph each week about what I have noticed in their collective writing (keeping them broadly within the parameters I set at the first of the class), and also point out the blogs that particularly impressed me. At the first of the semester, I tell them I will do this, and let them know, through a series of very unsubtle hints, that the posts I mention are the ones that got high marks that week. Over the course of the first three or four weeks, the spread of marks gets narrower and narrower as the students use the posts I mention as templates for their own writing. Our students are bright, and they want to do well. By showing them what I mean by doing well, they begin to seriously imitate the best. When they come and talk to me about how they can improve, I ask them if they have read the posts I have listed. When they tell me yes, I ask them if they notice a difference in what they are producing and what I have pointed out as being good work. They tell me yes, and then ask me what it is about the posts that make them better. I can (quite honestly) say to them that the really good posts make me go WOW!, and that is what they need to do. When they ask how that is done, I reply (again, quite honestly) “I have no idea, it just does”. They agree that it made them go wow as well, and then go away and try to make that happen in their work as well, and it often happens. “We are seeing peer-based learning networks where students are learning as much from each other as they are from their mentors and tutors (John Seely-Brown)”
I have had a single case of plagiarism in the five years I have taught this way. The students identified it (the student was using other students’ work and passing it off as his own), and were incensed that this would happen in their class. A delegation of students actually came to me demanding that the offending student be made to stand in front of the class and publicly apologise for what he had done. I told them not to expect that to happen any time soon. It was near the end of the semester, and the students didn’t even come back to the class again. Although this incident has had a few of my colleagues argue that we should protect students from this type of treatment, I tend to disagree. Although not a fan of Ayn Rand, I have to agree with her when she said “We can ignore reality, but we cannot ignore the consequences of ignoring reality.” I think that, too often, we try to protect our students from reality when that is exactly what they need to experience.
Finally, the public exposure opens them up to the likes of you and I -Professionals in the field who are always looking for good, interesting ideas that are presented in a well thought out format. In the past, my students have received favourable comments from around the world. One of my students received a scholarship to do a Masters degree at a prestigious university based on the blogs she wrote for my class. Someone commented on her work, and asked if she would like to collaborate with their research group, and when she explained that she was an undergraduate who was finishing up her degree that year, they asked if she would consider continuing her studies with them, all expenses paid. Unsolicited and unasked for, but welcome and appreciated. Not something you would get from having written an essay that only a single lecturer ever looks at (unless the work is double marked).
The requirement to comment on each other’s work is the other great learning outcome of using some SM tools for assessments.
I require my students to write five comments a week, bringing in fresh evidence each time to support the arguments they are making. This requires a significant amount of reading and thinking, and this is the one requirement that the students ask me to reduce every year. They are happy to write a blog post weekly, but feel that requiring them to make five comments makes for a heavy workload. I say, that’s what you’re here for.
Writing blog posts each week means that the students study a particular principle to a depth that I can be satisfied with in a senior undergraduate class. Having them comment on five of their peers posts means that they have to move out of their comfort zone, and engage in material that they otherwise wouldn’t. This satisfies me, as their teacher, that they have covered some breadth in the class.
However, I think the most powerful aspect of comments are the discussions and debates they spark. “It is better to debate a question without settling it than to settle a question without debating it (Joseph Joubert)”. I couldn’t agree more to a statement. They write, think and discuss matters in a lively, civil and scholarly manner. Everything I could hope for from my students.
As a learning tool, I have to say that blogging is one of the best. And social media blogging is far more powerful than blogging behind a firewall. In higher education, we deal with adults. We should be providing them with authentic experiences, and treating them like fully responsible adults. Helping them grow and develop in the real world in, for me, one of the most wonderful aspects of the work I do. I wish there were other who shared my excitement.
To understand. Simple enough.
I would be willing to wager (based on inside knowledge here – I’ve had the privilege of sitting on dozens of validation, revalidation, and quality assessment panels) that at least 80% of all learning outcomes use the word – understand.
We can define the word – perceive the intended meaning, significance, explanation, or cause of something. We teach it. We talk about it, or the lack of it in our students. We decry current study techniques that ignore it. It lies at the heart of what we do, bit can we operationally define it so we can measure it?
One of the first things I teach my students about the measurement of any psychological phenomenon, in the importance of operationally defining it. A good operational definition allows two observers to rate an interaction with high inter-rater reliability. Two people will return the same score when making the same observation. Sometimes you have good operational definitions, sometimes you get poor operational definitions, but you need one in order to have consistency in the measurement.
As an example, you might operationally define an interaction between two people as occurring every time their eyes met. This ignores any other kind of interaction (working together, body contact etc.) and only allows for a single item to be classed as an interaction, but you will get good, reliable measures when two raters observe the same event. It somehow feels phoney, but that’s what is done – I’ll come back to this later.
Understanding is a psychological phenomenon. It is something that takes place within an individual, and has to be inferred through some kind of observable behaviour. In formal education, we infer understanding through an assessment of some kind. There is a hierarchy of assessments that demonstrate understanding with true/false of multiple choice questions at the bottom, and oral examinations at the top (where you can probe for understanding until satisfied). The usual candidate for assessing understanding is an essay, either as coursework or an exam. However, what are we looking for when we say we are looking for understanding? What is our operational definition? How do we tell a novice marker what it is that we want them to reward?
Is what you are looking for in understanding the same thing that I am looking for in understanding? If we aren’t looking for the same cues to convey understanding in our students, which one of us is right?
According to Wiggins & McTighe, there is a matrix of 30 cells that have to be taken into account in order to represent understanding. These include such clear and precise statements as “a powerful and illuminating interpretation and analysis of the importance…” or “generally unaware of one’s specific ignorance”. Given that understanding is usually one of several attributes that are included in any good marking rubric (along with originality, synthesis, flow, grammar etc.), and each one of these attributes would be multifaceted, how many cells would we have to keep active in our working memory while reading through a piece of work. Good solid research tells us that we can only keep 5 ±2 items in our working memory at once – although I have met a number of lecturers who are particularly gifted, and insist that they can, and do keep all 1,987,392 cells of information necessary to perfectly grade a piece of work active in working memory simultaneously, in addition to their perfectly attuned scale so that they can differentiate between a 56% and a 57% on a piece of work – I’m afraid I must admit that I’m merely human, and 5±2 is my limit.
When something as ephemeral as understanding is pushed into an operational definition, the result is as phoney as defining human interactions solely as when two peoples eyes meet. Yes, you get good inter-rater reliability, but no one really believes that what has been carefully operationally defined captures real understanding.
And so we measure recall. If you can accurately recall the information that has been provided in the lectures and notes, you do okay. If you demonstrate understanding – whatever that is – you get higher grades. Might as well cut out all the reading and just go directly to MCQs.
I work in a university. Universities are in the business of teaching students (at least part-time). As a part of that teaching process, the learning that students undertake is evaluated, assessed, graded, and accredited. The purpose of this process is to provide a system whereby funders (parents and government), students, and consumers (workplace) can quickly evaluate the level of success attained by the person attending university.
Although the original purpose of the grading and accreditation was to provide an easily understood metric indicating the amount of learning that the student managed in university. Two fundamental problems have emerged from this simplified view of how the world works, 1) what are we measuring, and how do we measure it (the topic of a future post), and 2) how the focus of the student has shifted from learning to the metric (the grade, GPA, or degree).
One of the underlying principles in motivational psychology is the fundamental difference between intrinsic and extrinsic motivation. Intrinsic motivation (from a learning perspective) is the internally generated motivation that drives us to engage in learning and studying something out of our own, real interest. It is my grandson’s interest in dinosaurs, or my son-in-law’s interest in cars. My nephew’s interest in sporting statistics, or my friend’s fascination (or possibly obsession) with train engine names. These are interests that I don’t really understand, but drive the individuals to read, study, and understand information about their interest. My nephew has never been tested on his sporting stats knowledge, but I think he would pass a test with flying colours.
Extrinsic motivation is the motivation to do something because of some external driver. In the learning domain, it is learning something for some other reason that wanting to know it. Learning for a reward, or learning for grades, a degree, or a good GPA. It is not learning for the sake of learning, but learning to satisfy some other motivation.
The bedrock principle is that intrinsic motivation leads to real engagement, where as, extrinsic motivation does not. Extrinsic motivation leads to a level of engagement that will satisfy the extrinsic goal – which for some students means to pass, or just get a degree.
Our learning system has ended up producing many students who have real, intrinsic motivation, but not an internalised interest in learning a subject or simply acquiring knowledge, but an intrinsic motivation for acquiring grades. The learning becomes a side-effect of getting grades. The level of engagement is determined by the acceptable level of grades for the individual. Gaining the accreditation is the real goal.
And do we feed this? I would say that we do.
I read with interest Audrey Watters disappointment in the teachers and vendors at the ISTE convention playing for the gimmick in education. Something shiny and new that will catch the students attention for a few minutes (extrinsic motivation) and possibly get them to engage in learning (not likely). In Jones’ MUSIC model of student engagement, the ‘I’ stands for interest. He presents extrinsic motivation as situational interest, and intrinsic motivation as personal interest, and warns educators to be careful in their use of gimmicks (situational interest) to try to engage students. It doesn’t work unless the interest becomes personal (intrinsic). We had a discussion in our University LEG (Learning Enhancement Group) about the ‘I’ in Jones’ model, and I found it interesting that there was a discussion about the judicious use of gimmicks in teaching. The discussants were almost evenly split between those with a psychology background (hence, a level of psychological literacy) and those with different backgrounds. Those not from psychology were adamant that Jones was wrong to suggest that gimmicks don’t lead to student engagement in learning, while all those with a psychological background who kept pointing to the evidence that clearly states that the motivation has to be personal (intrinsic) or engagement doesn’t happen.
It was reported last week that students didn’t respond well (learning wise) to text messages sent to them with positive messages about education (extrinsic motivation). If we want to move our students from a focus on accreditation to a focus on learning, we need to find out how people learn. We need to move from the scholarship of teaching (with gimmicks and tricks) to the application of scholarship of learning.
Your post is getting at exactly what I was thinking.
There are two underlying concepts that you allude to in your reply. The first has to do with how we measure understanding. We are brilliant at measuring recall or recognition. We also have great, evidence based methods of maximising recall and recognition performance. Behaviourist and cognitive principles can show us how we can get excellent recall or recognition. This type of learning, and memory does play a central role in learning, can be measured by MCQs, T/F, short answer, and a range of other methods that can be easily automated. Excellent for use in learning that has memorisation and recall at the core. With those kind of tools, we can expect mastery, and get is, in a fairly simple manner.
However, how do we measure understanding, critical analysis or synthesis? This is the problem that has been at the core of educational research, and lies at the heart of mass education (or learning) today. How do we effectively measure understanding?
We recognise and know understanding when we see (or hear) it. However, it is not amenable to automation or quick measurement. It is not very responsive to quality assurance, and is difficult to accomplish in a heavily regulated environment. Vivas, in depth essays, well written blogs, long running projects, these can all measure understanding. However, these types of assessment take time, the second of the underlying concepts. The evaluation of real understanding takes time, and this is unacceptable in todays world. The evaluation of understanding also takes talent – as opposed to labor, which can be automated and reduced to clever algorithms.
As academics, this is what we should bring to education. As professionals in higher education, this is the real value added that we bring to the education system. Politicians and private enterprise are pushing an agenda (and winning the battle for hearts and minds) that would have all learning reduced to memorisation and regurgitation. This is the kind of learning that our students, parents, and administers are demanding – efficient, clean and measurable. Not the kind of learning that is of high value. Assessing understanding is messy, subjective, and not prone to measurement. But the most grievous of all sins that the measurement of understanding is guilty of is that it takes time. And time is money.
If I had the time, and the administrative systems allowed, I would have talked to my 52 students about what synthesis is all about, how you do it, and what it should look like, and then send them out to try again… and again… and again… until they either figured it out (with my support), or decided that this was something that they really couldn’t do, and found something more suited to their abilities (also with my support).
Unfortunately, in our system of education, this isn’t an option. As a result, I award a “C” grade and move on to the next batch of students and try again.
When we measure learning, we can measure it in two fundamental ways. Is the learner able to recall/demonstrate understanding/synthesise/critically evaluate – whatever – to a pre-set standard before the learning is finished (mastery), or is the learning graded (as in a gradient) and declared finished? The question I am asking is, if we don’t expect mastery in learning, what do we get?
The reason I am asking is because of a recent experience in my final year class. I asked the students to synthesise what they had learned about various topics for me. When I read their attempts at synthesis, I was sorely disappointed. At least 80% of them simply listed the various sub components they had covered, and submitted that as a synthesis. How do you evaluate something when your students don’t actually do what you ask them to. It was apparent to me that the students had never actually been taught what it means to synthesise information. In an ideal world, I would have returned them all, talked to them about synthesis, and then asked them to try again. I don’t live in an ideal world – I live in a world where the deadlines and forms of assessment are decided a year in advance, and no changes are allowed.
This leaves a marker in a quandary – do I mark them on what I asked them to do, or do I mark them on their attempt? I have talked to markers in other disciplines and in a number of institutions, and have found that I am not alone in this. What we do is look for somewhere that we can justify giving them marks. I know that we should all mark to a carefully constructed multidimensional matrix of a gradient of attributes, but as I have written before, humans simply can’t physically keep 30 or 50 or 80 cells (adequate on originality, or excellent on structure) in working memory while we read a document – although I have met a number of educators who swear they can – supermarkers (I humbly bow to their powers).
So, what do grades actually mean? What does a “C” grade on a synthesis exercise mean? I know that when my students write blogs, I can easily judge them on a couple of dimensions (critical evaluation of evidence and how well the information is presented). I also know that when I mark something more complex, like a final year project, the culmination of 18 months work, I am challenged to clearly articulate what a “C” grade actually means.
If I stick to my original problem, a synthesis blog, what does a “C” grade mean? Does it mean that the students failed to synthesise, but managed to list all of the content they were supposed to synthesise (that’s kind of what I did). How can I award credit for something that was asked for, but wasn’t done? What is the message that we give students when we give graded credit for their work? How does a poor essay differ from an excellent essay? Which dimensions are the critical dimensions that need to be evaluated.
The articulation (or lack thereof) of the critical dimensions of evaluation is the fundamental problem with blind double marking. It is almost impossible for two markers to agree on the subjective weighting given to the various dimensions that make up an average piece of work (grammar is more important that structure etc.).
So, what does a “C” grade say to the student, and what does that component say to a future employer? How has it become acceptable for us to say that someone is educated, they have a qualification, when they can get that qualification by only doing (write a synthesis) some of what the qualification says they need to do?
Shouldn’t an ideal system expect mastery? Shouldn’t we be able to say what a graduate has demonstrated the ability to do something? Shouldn’t we support a student in their trying until the succeed instead of giving them credit for their failed attempts and then pass them on?
I worry about what non-mastery really means…