Understanding and Time

John,

Your post is getting at exactly what I was thinking.

There are two underlying concepts that you allude to in your reply. The first has to do with how we measure understanding. We are brilliant at measuring recall or recognition. We also have great, evidence based methods of maximising recall and recognition performance. Behaviourist and cognitive principles can show us how we can get excellent recall or recognition. This type of learning, and memory does play a central role in learning, can be measured by MCQs, T/F, short answer, and a range of other methods that can be easily automated. Excellent for use in learning that has memorisation and recall at the core. With those kind of tools, we can expect mastery, and get is, in a fairly simple manner.

However, how do we measure understanding, critical analysis or synthesis? This is the problem that has been at the core of educational research, and lies at the heart of mass education (or learning) today. How do we effectively measure understanding?

We recognise and know understanding when we see (or hear) it. However, it is not amenable to automation or quick measurement. It is not very responsive to quality assurance, and is difficult to accomplish in a heavily regulated environment. Vivas, in depth essays, well written blogs, long running projects, these can all measure understanding. However, these types of assessment take time, the second of the underlying concepts. The evaluation of real understanding takes time, and this is unacceptable in todays world. The evaluation of understanding also takes talent – as opposed to labor, which can be automated and reduced to clever algorithms.

As academics, this is what we should bring to education. As professionals in higher education, this is the real value added that we bring to the education system. Politicians and private enterprise are pushing an agenda (and winning the battle for hearts and minds) that would have all learning reduced to memorisation and regurgitation. This is the kind of learning that our students, parents, and administers are demanding – efficient, clean and measurable. Not the kind of learning that is of high value. Assessing understanding is messy, subjective, and not prone to measurement. But the most grievous of all sins that the measurement of understanding is guilty of is that it takes time. And time is money.

If I had the time, and the administrative systems allowed, I would have talked to my 52 students about what synthesis is all about, how you do it, and what it should look like, and then send them out to try again… and again… and again… until they either figured it out (with my support), or decided that this was something that they really couldn’t do, and found something more suited to their abilities (also with my support).

Unfortunately, in our system of education, this isn’t an option. As a result, I award a “C” grade and move on to the next batch of students and try again.

What does Non-Mastery Mean?

When we measure learning, we can measure it in two fundamental ways. Is the learner able to recall/demonstrate understanding/synthesise/critically evaluate – whatever – to a pre-set standard before the learning is finished (mastery), or is the learning graded (as in a gradient) and declared finished? The question I am asking is, if we don’t expect mastery in learning, what do we get?

The reason I am asking is because of a recent experience in my final year class. I asked the students to synthesise what they had learned about various topics for me. When I read their attempts at synthesis, I was sorely disappointed. At least 80% of them simply listed the various sub components they had covered, and submitted that as a synthesis. How do you evaluate something when your students don’t actually do what you ask them to. It was apparent to me that the students had never actually been taught what it means to synthesise information. In an ideal world, I would have returned them all, talked to them about synthesis, and then asked them to try again. I don’t live in an ideal world – I live in a world where the deadlines and forms of assessment are decided a year in advance, and no changes are allowed.

This leaves a marker in a quandary – do I mark them on what I asked them to do, or do I mark them on their attempt? I have talked to markers in other disciplines and in a number of institutions, and have found that I am not alone in this. What we do is look for somewhere that we can justify giving them marks. I know that we should all mark to a carefully constructed multidimensional matrix of a gradient of attributes, but as I have written before, humans simply can’t physically keep 30 or 50 or 80 cells (adequate on originality, or excellent on structure) in working memory while we read a document – although I have met a number of educators who swear they can – supermarkers (I humbly bow to their powers).

So, what do grades actually mean? What does a “C” grade on a synthesis exercise mean? I know that when my students write blogs, I can easily judge them on a couple of dimensions (critical evaluation of evidence and how well the information is presented). I also know that when I mark something more complex, like a final year project, the culmination of 18 months work, I am challenged to clearly articulate what a “C” grade actually means.

If I stick to my original problem, a synthesis blog, what does a “C” grade mean? Does it mean that the students failed to synthesise, but managed to list all of the content they were supposed to synthesise (that’s kind of what I did). How can I award credit for something that was asked for, but wasn’t done? What is the message that we give students when we give graded credit for their work? How does a poor essay differ from an excellent essay? Which dimensions are the critical dimensions that need to be evaluated.

The articulation (or lack thereof) of the critical dimensions of evaluation is the fundamental problem with blind double marking. It is almost impossible for two markers to agree on the subjective weighting given to the various dimensions that make up an average piece of work (grammar is more important that structure etc.).

So, what does a “C” grade say to the student, and what does that component say to a future employer? How has it become acceptable for us to say that someone is educated, they have a qualification, when they can get that qualification by only doing (write a synthesis) some of what the qualification says they need to do?

Shouldn’t an ideal system expect mastery? Shouldn’t we be able to say what a graduate has demonstrated the ability to do something? Shouldn’t we support a student in their trying until the succeed instead of giving them credit for their failed attempts and then pass them on?

I worry about what non-mastery really means…

Graduates for the past

I went to a conference last week, and the focus was on how and why we need to have student centred learning. The talks were good, but it was the discussion that provided me with something to think about. I left to look up some of the numbers for myself – they are unbelievable.

The narrative goes something like this (you’ve read it before). Universities are providing the wrong skill set for today’s workplace. So, what are the implications for our students and our institutions?

In my last blog, I used Harold’s diagram that illustrates the difference between yesterday’s desirable skill set, and what needs to be added for today’s highly valued skill set.

To remind you, here are his identified skills (and some I have added) for yesterday and today.

Yesterday’s Desirables	Today’s Added Desirables
Obedience	Creativity
Conformity	Passion
Diligence	Initiative
Intellect	Willingness to take risks
Attention to Detail	Critical Debate
	Public Presentation

I have written that, as Universities, we are brilliant at producing graduates with the skill set on the left. However, we aren’t that great at delivering graduates on the right. As a result, there is a huge business in retraining workers (workplace training) to shift them from the left to the right of the spectrum. In the US, about £125 billion is spent annually to try to get workers where businesses need them.

Because we insist on clinging to an education that trains workers for yesterday’s world, the result is the following:

Sixty percent of graduates over the past five years are under or un employed. This is for Western Europe – in the UK it is 53% while the US is better at 52%. Research has shown that the first job that a graduate gets following university generally sets the level of employment they will enjoy for their entire career.

The graduate premium (the ££ worth of a degree) has fallen to £100,000 (not including student loans). This represents three years of average earnings in the UK. With evidence that graduates will pay up to £70,000 back for their student loans, that seriously reduces the graduate premium.

More worrying is that the premium does not apply the same to all graduates. For doctors and dentists, the premium is worth around £400,000. For communication studies graduates, the premium is closer to £5,000. For history and philosophy graduates, the premium drops to about £1400. And for creative arts students, the premium becomes a burden at about £15,000 (the red implies a loss). This is in comparison to those who enter the workforce at age 18.

Many of my colleagues tell me that the reason this has happened is because there are no jobs right now. Does this mean that there are no problems out there that need talented graduates to tackle? Of course not, but we don’t produce many of these – remember, we focus on conformity and obedience in university.

As we pretend that the world is not changing, the world is actually changing. That seems obvious, but not to the thousands of lecturers who dust off their powerpoint slides for another three hour drone – in between the grant writing and paper publishing (which is the real job).

I asked the panel at the end of our seminar what HE would look like in 20 years time. Because the panelists all work in HE institutions, the discussion took a dark turn for the worse.

It doesn’t have to be that way.

University: Preparation for???

Back in October, I wrote about the difference between talent and labour. I was inspired by Harold Jarche’s review of a NY Times article, and today I read another of Harold’s blogs along a similar vein. Harold writes for an audience that primarily focuses on work-based training, while I write from a HE perspective – not sure about my audience, I just write to get it off my chest.

Anyway, in his blog, Harold contrasts the values that traditional industries looked for in workers - obedience, diligence, and intellect – with the skills desirable in todays workplace - initiative, creativity, and passion. In the workplace, this maps closely to the labour and talent continuum.

From http://www.jarche.com/2013/05/the-new-work/

This begs the question for us in HE to consider: Why are we preparing our graduates for producing?

As Steve Wheeler observed last week:

We live in uncertain times, where we cannot be sure how the economy is going to perform today, let alone predict what kind of jobs there will be for students when they graduate in a few years time. How can we prepare students for a world of work that doesn’t yet exist? How can we help learners to ready themselves for employment that is shifting like the sand, and where many of the jobs they will be applying for when they leave university probably don’t exist yet?

To prepare our students for yesterdays jobs is doing them a disservice, and yet, that is exactly what we are doing. In March I wrote about conformity and creativity in education. Our formal systems are all about conformity (and obedience, and diligence, and attention to detail – all highly valued in yesterdays workplace) and in the process, we go out of our way to stifle creativity. I was in a Teaching and Learning Committee meeting a few months ago with the topic being warmly debated being the penalties to be imposed on students who exceed word limits on their assignments. This wasn’t a team meeting or a departmental T&L committee, but a regulations meeting for the entire University. One of my colleagues commented that he wasn’t willing to read “War & Peace” when a student turned in an assignment. I innocently asked the question, “What if they actually produced War & Peace as their work?” The response is telling (I think) – “No chance of that with our students.” I have to agree – no chance of that. There’s no room in the syllabus for creativity, passion or initiative, only obedience, diligence, conformity and attention to detail.

We focus on the traditional, time honoured form of teaching. I’ll tell you what I know, you memorise it, and in a few weeks, I’ll ask you to regurgitate it back to me. This will either be in the form of a set essay question (I will encourage your creative juices by giving you the choice of two topic to choose from), an unseen essay exam (as before, but with no resources), or an unseen tickbox MCQ or short answer exam (we have to examine on all the material). I find that whenever I talk about this form of teaching, lecturers go red in the face and indignantly say to me “I expect them to bring something into their writing that I didn’t say in class!”. Okay, regurgitate, with a little bit extra. Conformity, obedience, diligence, and attention to detail – all virtues prized in producers, not thinkers.

Why? It is because we have so much material to cover.

The tyranny of content. I have written about this topic on a number of occasions, and yet, content expands just like The Blob. Since the first time I wrote about content almost two years ago, according to the pundits, information had doubled twice (approaching thrice). This means that the lectures scheduled for this autumn will have to contain twice as many slides, and the lecturers will have to go through them twice as fast as they did last year. There is no room for creativity or thinking when there is just so much that has to be learned (memorised).

Steve quoted Etienne Wenger in his blog last week  ‘If any institutions are going to help learners with the real challenges they face…(they) will have to shift their focus from imparting curriculum to supporting the negotiation of productive identities through landscapes of practice’ (Wenger, 2010).

When are we going to start seeing these vital changes happen? When will we have institutions changing to embrace tomorrow instead of clinging to yesterday?

Our unwritten contract with society has been breached – by us. Donald Clark (I think convincingly) suggests that it has been our obsession with grades and qualifications that has done it. With up to 60% of Western Europe’s graduates (and similar numbers from North America) from the past five years either un- or under-employed, we don’t have a shortage of qualifications, we have a shortage of learning.

We have such a rich history of teaching and learning in higher education. We must look ahead and figure out how we can change the system to return to a place where learning lies at the heart. We have wasted too much human resource by our inability to move on from what we have succeeded with in the past. Tomorrows world needs something different, and if we don’t deliver it, someone else will.

Higher what?

I read Harold Jarche’s review of Marina Gorbis’ new book, and was thinking about what to write about it (it struck a chord) and then Clark Quinn came up with something brilliant. It was his comments about our unique human skills and specifically our abilities for sensemaking, and  novel and adaptive thinking that hit me. In a nutshell, these are our higher order thinking skills. Higher, as in higher education. Higher as in the manipulation of the abstract in order to find emergent patterns. Higher as in taking the mundane and creating something new. Higher as in bringing together disparate strands and synthesising information to produce a new, emergent thing (property, thought, theory etc.). This is what sets us apart as different. We can do these things, and we can do them well. Even at the most basic level, we learn to decipher recognisable thought patterns from a series of sounds (speech), and we learn to decipher thoughts from abstract symbolisation (writing).

Early in our history, we figured out how to fashion tools that would augment our physical abilities. We learned to hunt with implements, farm with tools, and lift and move with technology. The augmentation of our physical abilities has allowed us to build and create on an unprecedented scale. We are building a new arts and innovation centre, Pontio, at Bangor University.

New arts and innovation centre at Bangor University, scheduled to open in 2014.

As I watch the building grow from my office window, what I find amazing are our abilities to lift and move the hundreds of tons of materials and carefully put them in place. We can do this in a relatively short time period because we rely on tools that amplify our natural lifting and moving abilities.

In the last 50 years or so, we have developed some equally impressive tools to amplify some of our thinking abilities. Not the higher order abilities, but many of the routine thinking abilities we possess. We have machines that can do routine and complex calculations, memorise and regurgitate with impressive speed and accuracy, and with storage and searching facilities that are immense and fast. And, yet, we don’t have machines that can do what makes us really special: synthesise and create.

The sad thing is that in our higher education institutions, we still focus on the routine calculations, the rote memorisation, and the regurgitation skills that we needed in the pre-computer age. I hear my colleagues say to me that if our students don’t learn all the content, and memorise all the ideas, they couldn’t function well if all the computers in the world suddenly stopped working. I wonder if construction workers worry about their buildings remaining unfinished if all the cranes in the world suddenly stopped working.

Instead of challenging our students to sharpen the skills that only we (humans) possess, we assemble them in massive groups and lecture them, bringing them back together several months later to go through an examination exercise; the routine memorisation/regurgitation exercise. Is this the best we can do?

By the way, our beautiful new Pontio building is going to have a massive lecture theatre (over 500 seats), accompanied by a number of smaller (200+) lecture theatres so that we can perpetuate and expand our carefully honed memorise/regurgitate routines. How sad.

Caution – Shallow waters all around us

Roger Shank’s blog post yesterday struck a note with me when I read it this morning. As I was reading the BBC’s article about Oxford’s Future of Humanity Institute, I thought about what Roger was saying, and how the thinking questions and real issues seem to have been displaced by the vacuous.

In HE, we teach students how to answer questions with the right answer. John, at Learner WebLog was comparing the c and x versions of MOOCs, and why the xMOOCs attract so many more people. In his analysis, John states one of the factors as being that xMOOCs are easier, and writes

xMOOCs are much easier compared to cMOOCs.  This is grounded on that in xMOOCs, the instructors would have done most, if not all of the ground work necessary for teaching and learning for the learners.  What the learners are normally expected to do would be to consume the knowledge transmitted or broadcasted to them, and to confirm their understanding of the concepts through repeated quizzes or assignments.  This requires certain perseverance from the learners, though it is possible to achieve a high or perfect score in test, assignments and examinations through drills, repeated practice, as is common in a rote learning scenario.   The use of standard answers in the case of multiple choices, true/false, or short case scenarios, could all be checked with automated grading or assessment software.  For peer assessment, these are done in a closed manner, with the merits of “protecting” the learners from being “criticised” in public, but the demerits of being critiqued by only a few participants (5 other peers) in the whole evaluation.  Nevertheless, this seems to be well accepted as a way to assessment in the xMOOCs, as that might be the only feasible and reliable way to assess students in an institutional environment, without overly involving the professors in the assessment.

People want the right answers. We want to be correct, and don’t want the effort of having to figure out something that isn’t black and white. As my students finished up my module, where I use a connectivist and constructivist approach to learning, one of the reoccurring themes in their reflection about the beginning of the module was along the lines of what is Jesse looking for, or is this going to get me a good grade, or how can I learn anything without an expert telling me what I need to learn. Having the freedom to explore a topic, with no right or wrong answers was terrifying. They couldn’t understand that I wanted to see their thinking, their evaluation, their critical approaches. However, by the end of the semester, they loved it.

As I (an many of you) talk about the faults of the educational system, the zombification of learning, the factory foundations, the memorisation/regurgitation model, am I out of touch? Donald Clark tells us that the social contract between HE and society is broken, and uses as evidence the graduate unemployment and underemployment rates, the riots in London a couple of years ago, the occupy movement, the disillusionment that industry leaders have in graduates today. The list could go on, and on, but is anyone listening? There are good people who talk and think about the shallowness of todays educational system, with its fixation on content. There are good people who are doing what they can, within their sphere of influence, to change things. But, is there any measurable impact? MOOCs have become the darling of education today, and yet, the xMOOC model seems to be the only game taken seriously. The most powerful educational institutions in the world have abdicated their responsibility for learning and have opted for MCQs.

This leaves us with more academic articles being published about snowboarding than about real threats to human existence. It leaves us with the endless pursuit of the right answer and the good grade. It leaves us with 70% of graduates in the UK this year about to be awarded a top degree (2i or 1st). It leaves us in the shallows with everyone leaning over the edge of the boat examining their reflection in the water to see how well their complexion is holding up.

Learning and Grades

During this spring semester, Emma, one of my students, has been discussing grading as a part of our educational system, and what effect grading has on learning. Last week, my students pulled their thoughts together and produced synthesis blogs. I thought I’d share some of Emma’s insights.

Grades lead to conformity. Although Susanna (another of my students) argues that conformity in education can be a good thing (she hasn’t convinced me – but my evaluation of her work has remained unbiased — I hope), I agree with Sir Ken Robinson that stifling creativity in education has been a terrible thing. Trudi (yet another student) explored the impact of the factory model on education, and how conformity was one of the principle aims of mass education at its inception. Emma asks the questions that I have posted on a number of occasions “Do students come to university to learn or do they come to get a degree? Does a high class degree constitute a high level of learning?” I think she successfully argues that current HE is about figuring out the hoops that you need to jump through to get high grades. Jumping through hoops allows no creativity, it requires complete conformity.

Her second point is that grades gamify education. The focus becomes the grades rather than the learning. Games have rules, and rules lead to conformity. If you don’t follow the rules (learn what the teacher wants you to learn) you don’t get the points (your grades suffer). Working to the grades means students can memorise and regurgitate what is expected of them.

She then goes on to look at the research into the validity and reliability of grading. Do grades measure learning? I lover her quote from Race (2009) where he argues (I think effectively) that a written “exam does tell you is how well learners write, not how well they have learnt. What is measured is not necessarily learning, but neatness, speed and eloquence of learners’ writing”. An important skill, but is that learning?

The only disappointment I had was her conclusion that grades are here to stay, because that is the system.

On my current CV, there is no reference to grades. What I have listed are my accomplishments – not someone’s evaluation of my accomplishments (a grade). Authentic assessment would allow a student to produce something that they could refer to on their CV. I hope that my students list their blogs on their CVs. These are real accomplishments that have provided me with much to think about over the past 10 weeks.