What does Non-Mastery Mean?

When we measure learning, we can measure it in two fundamental ways. Is the learner able to recall/demonstrate understanding/synthesise/critically evaluate – whatever – to a pre-set standard before the learning is finished (mastery), or is the learning graded (as in a gradient) and declared finished? The question I am asking is, if we don’t expect mastery in learning, what do we get?

The reason I am asking is because of a recent experience in my final year class. I asked the students to synthesise what they had learned about various topics for me. When I read their attempts at synthesis, I was sorely disappointed. At least 80% of them simply listed the various sub components they had covered, and submitted that as a synthesis. How do you evaluate something when your students don’t actually do what you ask them to. It was apparent to me that the students had never actually been taught what it means to synthesise information. In an ideal world, I would have returned them all, talked to them about synthesis, and then asked them to try again. I don’t live in an ideal world – I live in a world where the deadlines and forms of assessment are decided a year in advance, and no changes are allowed.

This leaves a marker in a quandary – do I mark them on what I asked them to do, or do I mark them on their attempt? I have talked to markers in other disciplines and in a number of institutions, and have found that I am not alone in this. What we do is look for somewhere that we can justify giving them marks. I know that we should all mark to a carefully constructed multidimensional matrix of a gradient of attributes, but as I have written before, humans simply can’t physically keep 30 or 50 or 80 cells (adequate on originality, or excellent on structure) in working memory while we read a document – although I have met a number of educators who swear they can – supermarkers (I humbly bow to their powers).

So, what do grades actually mean? What does a “C” grade on a synthesis exercise mean? I know that when my students write blogs, I can easily judge them on a couple of dimensions (critical evaluation of evidence and how well the information is presented). I also know that when I mark something more complex, like a final year project, the culmination of 18 months work, I am challenged to clearly articulate what a “C” grade actually means.

If I stick to my original problem, a synthesis blog, what does a “C” grade mean? Does it mean that the students failed to synthesise, but managed to list all of the content they were supposed to synthesise (that’s kind of what I did). How can I award credit for something that was asked for, but wasn’t done? What is the message that we give students when we give graded credit for their work? How does a poor essay differ from an excellent essay? Which dimensions are the critical dimensions that need to be evaluated.

The articulation (or lack thereof) of the critical dimensions of evaluation is the fundamental problem with blind double marking. It is almost impossible for two markers to agree on the subjective weighting given to the various dimensions that make up an average piece of work (grammar is more important that structure etc.).

So, what does a “C” grade say to the student, and what does that component say to a future employer? How has it become acceptable for us to say that someone is educated, they have a qualification, when they can get that qualification by only doing (write a synthesis) some of what the qualification says they need to do?

Shouldn’t an ideal system expect mastery? Shouldn’t we be able to say what a graduate has demonstrated the ability to do something? Shouldn’t we support a student in their trying until the succeed instead of giving them credit for their failed attempts and then pass them on?

I worry about what non-mastery really means…


Behaviourism and Learning

I know that a number of psychologists will tell you that behaviourism and learning are just two names for the same thing, and they might be right is some way, but the learning I am talking about isn’t based on rats running a maze, but students learning higher thinking skills. After all, higher thinking skills is what the higher stands for in Higher Education. The rats running a maze is where the behavioural principle come from that I want to focus on. And for those who thought there was a cognitive revolution in psychology, behaviourism is still very much alive and kicking (as it should be – even though I am a party to the revolution). The principles are still as valid today as they were when they were articulated 50 years ago.

So what does behaviourism have to say about learning (as in education)? Actually, it can tell us quite a bit. I have been reading and writing about assessment this past year, and it astounds me that the assessment methods, for as much as we talk about innovation, are still very much based on tradition. Given how much we know about how we learn and master a skill, why are assessments still all done, basically, the same way?

Let me explain. Behaviourist interventions are the best interventions we have for changing behaviours (I’m not talking about a philosophy here, but about what the evidence says). The basic principle for a behaviourist intervention is as follows. Identify a behaviour that needs changing, isolate that behaviour, take a baseline measure of the behaviour, introduce the intervention, and then measure the effect of the intervention. It can be a bit more complicated than that, but not much. You could think of education and learning as changing thinking and behaviours. After all, that is essentially what we are trying to do. Looked at through a behavioural prism, we don’t do a very good job of changing behaviours and thinking. It isn’t that the students (our subjects) aren’t willing, it is just that they struggle to figure out what we want.

What I mean by that can be understood if you think about a student submitting an essay for grading. You might say that innovative assessment means that an essay is only one tool from a large toolbox that is available, however, the point I am making applies to most what is available and used. In an essay, we expect that the student will have a well structured argument, use strong sources, show evidence of critical thinking, throw in a bit of originality, write with an easy to read style, use proper spelling, punctuation and grammar, and get the content right. That means that we are evaluating (at least) nine dimensions on a single piece of work. We expect the student to produce a multidimensionally superb piece of work that we then take in for marking. We, in as little time as possible (given that there are 183 sitting on your desk due back next Tuesday), become an expert judge (a future blog on this concept) on a hypercomplex problem, providing a simultaneous evaluation on the multifaceted dimensions, and awarding an appropriate level of credit for the work. And then we wonder about the lack of reliability between markers

So much for isolating something that we need to change, introducing an intervention, and then measuring the outcome of the intervention. Why can’t we design a higher level learning environment that isolates the skill or content that is targeted, and then introduces an intervention that continues until a mastery level of achievement is reached before introducing something more complicated.

In my Science of Learning module, I focus the students on the providing evidence from an acceptable source for their blog entries and incorporating that evidence into a well structured argument. Two higher thinking skills that the students practise over and over (at least 14 times) during the semester. One of the criticisms that I faced the first time I taught this way is that final year undergraduates shouldn’t all be able to get top marks in a class, there needs to be more spread recognising their varying abilities. However, I asked, in our examination board, if the students meet the criteria that I set, and they meet it well, shouldn’t they receive full credit for having done so? The exam board agreed with me. They also agreed that the learning outcomes ( providing evidence from an acceptable source and incorporating that evidence into a well structured argument) for the module were more than appropriate for a final year class. Just because the bulk of the students master it shouldn’t make their accomplishment any less.

I think it is a shame that this kind of learning is so rare in education. It doesn’t have to be.

Learning and Education

When did education stop being about learning and turn into a performance art?

I was reading over some of my students’ blogs from last semester, and one of the things that jumped out at me was their observation that education was about grades, degrees, and getting ahead and not about learning (they weren’t happy about it – often pointing out that this is what is wrong with education today).

I blogged last year about how Bjork talks about the conditioning cycle that moves both students and learners into a self reinforcing cycle of performance and reward. Students are rewarded for doing what the teacher wants (high grades) and teachers are rewarded for increasing the number of students who achieve high grades (promotion opportunities, institutional acclaim). This becomes a virtuous (vicious) cycle of mutual rewards as students learn to perform (who said passing a test had anything to do with learning), teachers recognise the performance with academic currency (grades) and institutions reward “good” teaching with recognition and praise. Who is fooling who?

Jack Rogerson (one of my students) blogged about student cheating and why. He noted work by Dweck & Vandewalle who identified performance goal oriented students as:

  • Maladaptive Students – Quickly become disillusioned with tasks and tend to discourage themselves from developing their academic abilities/skills. They instead focus their attention on the opinions of others – they are mindful of negative judgement and are therefore more likely to resort to cheating as means of maintaining a positive image of capability amongst their peers.

It is all about appearances.

I believe that there are many students who start their studies actually excited about learning, but eventually, most find themselves caught up in the performance and reward cycle.

For me, this is one of the damning features of lectures. I will stand up and tell you something that I think you should know (and/or record it as a podcast and post it to the world), and then, in the name of assessing your learning, ask you about what I told you. The better you are at fetching the information (including some tidbit that I didn’t actually share with you), the higher the reward you will receive.

I wonder what Socrates would think of our civilised approach to learning today?

It doesn’t have to be that way. Present something both interesting and useful, care about the students success, empower them to direct their own learning experience, and help them believe that they can succeed. Provide learning tasks that allow students to match their ability with your expectations, and then reward real success, not a momentary performance.

We have the know how and the tools to liberate the learning experience. We can really have students centred learning – for which lecturing is the antithesis – at every learning  opportunity. Using an information abundance model to underpin learning design, I have scaled student centred learning up to 60+ students at a time. We don’t need to have seminars and discussion of >10 students to have a real learning experience, it can be available now with reasonable resources.

Given what we have available and what we can do now, I despair at the cost of inertia.

Teaching with Podcasts – A Great Success Story

Teaching labs for statistics classes is one of those labour intensive, not very much fun, teaching jobs. In Bangor, we enjoy 300+ students a year studying psychology, and teaching the students to use SPSS to analyse data is one of those difficult and thankless tasks no one really wants.

For many years, we divided our numbers up into groups of about 50, and then repeat taught six sessions for 90 minutes for about eight weeks each semester over three semesters. Even with an average of three postgrads helping out in each lab session, the results were never very positive – the most able were bored, the less able were lost, and the middle felt pushed, but kind of got it.

My colleague, Mike Beverley, was responsible for teaching the labs across both the first and second years while I taught the first year classes. The topics covered ranged from simple data entry to complex ANOVAs and factor analyses. Students were set assignments, analysed data, and turned in regular work. The experience was not very enjoyable for either the students or the teachers. Our satisfaction ratings from the students was low when it came to the labs (“Burn every copy of SPSS on the planet!” was typical). We needed a new model.

We initially decided to podcast Mike’s first session each week, and then he could ensure that the presentation to the students was at least consistent, and he didn’t have to repeat himself endlessly. Podcasting was new then (spring of 2005), so we were just trying things out. At first we recorded audio podcasts, but after a few weeks, we recorded the screen capture for the students. By chance, we stumbled onto a model that really worked. In the labs, we would usually provide about five minutes of instruction, and then let the students work at it for a few minutes before introducing something more. As a result, all of our podcasts were about five minutes long, demonstrating how to carry out a procedure with a voiceover. The initial fumbling about was successful enough that we decided to prepare podcasts over the summer to cover every topic we taught for use the following year.

In the Autumn of 2005, we scheduled the labs, employed the postgrads, and demonstrated the podcasts to the new, incoming students. To our surprise, no one ever came to another scheduled lab. I fib here – there were about six students who insisted on coming every week. After about three weeks, we rolled the six into a single session, and let the rest of the students know that we were only going to be in the lab for that single 90 minute session. If they had question, they needed to come along then.

The students learned SPSS – better than they had in previous years. Their feedback in the module evaluations was uniformly positive about learning SPSS that way, and we changed the way we did things (we still use the same basic model six years later).

Cost Savings

The savings from this have been great for the Department. Teaching the traditional labs went something like this:

  • 9 hours/week per instructor
  •         4 instructors = 36 hours/week
  •    8 weeks teaching across 3 semesters
  •    864 hours/year of instructional time
  • Stats support surgeries for 3 hours/week across 15 weeks involving 2 or 3 people
  •    Approximately 75 hours/year
  • Total of about 940 hours year teaching and supporting stats

Using podcasts for instruction, the cost went someting like this:

  • 22 weeks with 1 hours of support available per day (1 person currently)
  •      110 hours
  • Savings of 830 hours

Podcast Development

  • Estimate about 3 – 4 hours per lesson
  •       2 or 3 podcasts per lesson
  •       23 lessons
  •    70 podcasts in total
  • About 100 hours in total to make the initial podcasts

Bottom Line

Massive savings overall (830 hours less development time), and the podcasts are reusable. The instructor was happier (for a time), the students were happier. The students have the podcasts available throughout their entire undergraduate programme so they can refer to them anytime. They have control over their learning,and use the labs when they choose to do so. A real win – win solution!

The single biggest problem has been the updates of the programme (SPSS). This has meant that we have re-recorded the podcasts twice in six years – not too bad an investment given the long term benefits.

Success Secrets

I think there were a few things we did (and continue to do) right when we made the teaching podcasts. They were:

  • Have an expert in Stats & SPSS teaching make the podcasts (at least the first time)
  • Keep the podcasts short (5 – 10 minutes)
  • Don’t obsess about quality
  • Be prepared to release updates quickly if there is a need for clarification

In the area of teaching and learning, given the promises of efficiency and performance that technology has alluded to over the years, it is nice to see something that turns out to really work – along with some quantifiable evidence to illustrate just how well.

We started this way back in 2005 – I just haven’t put this out there since. I have presented it at a couple of conferences, but not put it out there where anyone could refer to it, so here it is.

Guaranteed Educational Outcomes

One of my students recently blogged about a mismatch between graduate skills and industry requirements in the gaming industry. I shook my head as I read that, although the industry is crying out for programmers, fewer than 12% of the graduates from specialist gaming degrees are employed six months following graduation. Fifty eight percent of employers in the industry say that specialist graduates have the skills necessary upon graduation (rising to 71% in larger firms). We have become a dishonest broker between incoming students and industries looking for employees. At one end, our market is made up of 18 year olds who are interested in the student experience while at the other end, our market is made up of employers who are looking for a set of skills. We make the noises that are satisfying to both ends, and end up letting everyone down.

Why not guarantee the skills of our graduates? Why not put an iron-clad guarantee on what a graduate of our department/school/institution can do, backed by our reputation (some would say we already do that) and some cold hard cash?

Let’s look at how that might work. I guarantee that a student who graduates from my degree programme with a 2i (B range average) can write at Level X, speak at Level X, manipulate numbers at Level X, engage in critical thinking at Level X, and evaluate evidence at Level X. If you are not satisfied after one year in the workplace with the quality of my graduate (barring any unforeseen circumstances), I will reimburse you the full year salary that you paid my graduate.

Different skill levels would be attached to different programme outcomes. Students could enrol on the course knowing that it would be tough, but that there would be a guaranteed job at the end, and employees could hire new graduates knowing that if they did not get what they were expecting, their money would be refunded. Intake into the programme would be determined by the number of guaranteed placements available at the end.

We could do this today — so what is stopping us?