How can e-learning help with student feedback?

I attended a teaching and learning meeting in one of our academic departments recently when someone asked if they could use technology to improve their feedback to students. Four possibilities sprang to mind.

Online marking  – As an associate lecturer at the OU I have to use online feedback via standard forms and document mark-up (i.e. comments in Word) is obligatory. After several years I now have a ‘bank’ (personal collection) of comments I can draw on to quickly provide rich personalised feedback. Moreover the OU uses ‘rubrics’ (marking schemes) to structure feedback and make sure it is aligned to the learning outcomes. This improves the efficiency and effectiveness of marking and honestly I’m not sure I could manage without this approach now. Here in UCL many colleagues use Moodle Assignments to allow markers to bulk upload markers’ annotations on files of student work, and to set up rubrics and other structured marksheets.  GradeMark, part of Turnitin, has a particularly convenient marking environment. Its unique selling point is customisable sets of frequently-made comments which provide online marking with drag and drop and general online comments. Comment ‘banks’ are available at a click and a drag, and can be shared across programmes and they can themselves be linked to rubric structures.

Self-assessment – Perennial favourites in UCL student surveys are diagnostic and self-assessment quizzes, usually developed in Moodle’s Quiz, which despite its frivolous name is actually a very sophisticated assessment and feedback tool. Moodle quizzes offer different question types, including multiple choice, gap fill, and drag-and-drop,  all of which can automate giving feedback, based on the settings tutors choose.  For example, feedback could be given immediately a question is answered, or it can be deferred until after the quiz is completed. Students appreciate the chance to check progress and get focused feedback based on the answer they chose.  While writing good questions and feedback takes thought and care , technically quizzes are comparitively easy to set up in Moodle. The trick is to provide good, differentiated feedback, linking to remedial or additional materials that the student can look at straight away. In Moodle these links could be to documents, items on the electronic reading lists, Lecturecast recordings, YouTube videos and so in as well as simple texts and images. Questions can be imported from Word using a template, allowing rapid quiz authoring without an internet connection, and even the Matlab GUI has been used to automatically generate mathematical question banks for later import. As an alternative UCL has also had some success using PeerWise enabling students to design their own multiple-choice questions (MCQs).

Audio and video feedback – One interesting feature of GradeMark is the facility to provide students audio feedback. Staff at UCL have been experimenting with audio feedback for several years, adding live audio comments to text documents or forum posts, for example. The rationale is that feedback is richer, more personal, more expressive of different emphasis, and there is more of it for the amount of time spent. Since it also tends to be less concerned with particularities of grammar, spelling, etc, some markers may want to combine with word-processed annotations. An extension of this is to make a single recording giving general feedback to an entire cohort on a given piece of work. And an extension of this idea is to create simple narrated screencasts using Lecturecast personal capture (desktop recording) to record worked examples for generic assessment and exam feedback. This approach has been tried in at least one department with positive results.

Peer assessment and MyPortfolio – Technology can of course provide whole new ways to enable group assessment and the development of rich personal portfolios, for example using the increasingly popular UCL portfolio and presentation environment MyPortfolio.  For an excellent introduction, have a look at the recent UCL case study Making history with iPads, peer assessment and MyPortfolio.

Further reading

Image “Got Feedback?” by Alan Levine https://flic.kr/p/nKPbtE

 

Grounding the Cloud

dropbox-blueprint-blogIn April, Vicki Dale and Clive Young, E-Learning Environments carried out a survey with UCL teaching and support staff about their use of cloud-based tools for teaching, research, administration and personal use.

We wanted to find out if ELE should support any of these tools at all and if so to what extent, given the external and rather fluid nature of such services. Over 200 staff – mostly academic colleagues – responded to the survey from across UCL. We found evidence of use of a wide range of external cloud-based tools, many used personally, but also to support teaching, research and administration.

Although overall use was not as high as we had anticipated, some specific tools were used quite a lot. Skype was the most personally used technology (52%) while for teaching and research Dropbox was top (both 39%) and more than half of respondents had used Doodle for admin.

The top tools were:

Purpose Most used single tool Average use (across teaching, research, admin and personal) Mostly used for …
Filestore/file sharing Dropbox 43% Personal (48%)
Video/voice calls Skype 37% Personal (52%)
Event organisation Doodle 27% Admin (54%)
Professional networking LinkedIn 21% Personal (36%)
Social video sharing YouTube 21% Teaching (33)
Short message broadcasting Twitter 17% Personal (24%)
Social networking Facebook 17% Personal (42%)
Online office tools Google apps 16% Personal (17%)
Web conferencing & webinars Skype 16% Research (19%)
Instant messaging SMS texts 13% Personal (40%)
Filestore/file sharing Dropbox 43% Personal (48%)
Video/voice calls Skype 37% Personal (52%)
Event organisation Doodle 27% Admin (54%)
Professional networking LinkedIn 21% Personal (36%)
Social video sharing YouTube 21% Teaching (33)
Short message broadcasting Twitter 17% Personal (24%)
Social networking Facebook 17% Personal (42%)
Online office tools Google apps 16% Personal (17%)
Web conferencing & webinars Skype 16% Research (19%)
Instant messaging SMS texts 13% Personal (40%)

 

It seems though that many of the tools are still only being used by enthusiasts, especially for professional purposes. This group could broadly be regarded as the self-starting ‘innovators’ and ‘early adopters’ (as identified in Rogers’ innovation curve), who are keen to experiment with new technologies without organisational support or encouragement.

Although not really surprising, this interpretation runs slightly counter to the popular notion that such tools are largely self-supporting and need little guidance.  Nevertheless there may be a silver lining in this metaphorical cloud. Some tools are used much more for personal use and people are comfortable in using them. There could be the potential to adopt some of these tools for educational purposes.

We concluded that, firstly despite the hype around cloud-based tools and services, they are actually much like any other technologies. To increase their take-up, staff need to be provided with adequate support to work out what tools are appropriate to use and how they may be used in a professional context. Secondly, tools that we (as learning technologists) regard as ‘cool’ and cutting edge, may be seen by non-enthusiasts as unstable, unsupported and risky.  We may have to rethink what tools and services we provide centrally. How can we provide the functionality of cloud-based tools that our colleagues evidently want for teaching, research and administration, but in a more supported and stable low-risk environment?

Image: dropbox.com

Webinar: UCL working with the new change agents

webinarClive Young (ELE) and Stefanie Anyadi (UCL Division of Psychology and Language Sciences) led an ALT webinar today on work UCL has been doing with our community of teaching administrators (TAs).

We described the now-completed JISC Digital Department project that supported these staff in developing their digital literacies and in working more systematically and strategically with them as change agents. This had led directly to the establishment of our supported programme leading to the Certified Membership of the Association for Learning Technology (CMALT). We also introduced the E-Learning Champions initiative and explained why we had included TAs to work in partnership with academics and ELE staff. Although very much a work in progress this has proved effective and has already helped benchmark e-learning activity, develop local plans across two of our schools and has led to the emergence of active faculty-level e-learning groups.

The slides and recording are available on the ALT Repository at http://repository.alt.ac.uk/2351/

Webinar: UCL working with the new change agents

Clive Young (ELE) and Stefanie Anyadi (UCL Division of Psychology and Language Sciences) led an ALT webinar today on work UCL has been doing with our community of teaching administrators (TAs).

We described the now-completed JISC Digital Department project that supported these staff in developing their digital literacies and in working more systematically and strategically with them as change agents. This had led directly to the establishment of our supported programme leading to the Certified Membership of the Association for Learning Technology (CMALT). We also introduced the E-Learning Champions initiative and explained why we had included TAs to work in partnership with academics and ELE staff. Although very much a work in progress this has proved effective and has already helped benchmark e-learning activity, develop local plans across two of our schools and has led to the emergence of active faculty-level e-learning groups.

The slides and recording are available on the ALT Repository at http://repository.alt.ac.uk/2351/

UCL E-Learning Champions – one year on

Clive YoungThe UCL E-Learning Champions initiative is actually now a bit over a year old – we launched on Valentine’s Day in 2013. At the Summits & Horizons event on Monday we took the chance to review progress so far.

We have certainly grown over the last year, we now have over 130 members. Nearly all UCL departments and divisions are represented and although there is work to be done (not everyone knows who their Champion is!) in some areas the initiative seems remarkably well established.

I opened the session by reminding the sizeable audience that the Champions were key to our ambitious institutional target to develop departmental e-learning plans. Pairs of departmental E-Learning Champions – one academic and one teaching administrator (TA) – had been identified to help develop local e-learning statements articulating departments’ expectations and priorities, highlighting good practice, and identifying support needs.

Fourteen months into the implementation plan, this type of engagement with the champions has been excellent. Champions are well embedded in UCL’s learning technology strategic and support initiatives and are now beginning to be regarded as an important group of change agents, with the support of ELE.

This approach has already helped ELE benchmark e-learning activity, develop local plans across two of our schools and has led to the emergence of faculty-level e‑learning Champion groups. During the event our three E-Learning FacilitatorsJess Gramp (BEAMS)Natasa Perovic (SLMS) and Mira Vogel (SLASH) explained how the Champions had helped with a wide range of local initiatives and projects, in areas such as Turnitin, multimedia group work and enhancing Moodle provision.

The Champions’ role is also evolving. Mira reported that SLASH Joint Faculty Departmental Teaching Committee Chairs had been asked to ensure that their departmental E-Learning Champions were either members of the DTC or had a direct reporting link to it. SLASH has also convened an E-Learning Forum to discuss the issues of the day, identify support needs and plan future directions. Likewise in BEAMS there are now regular E-Learning Champions Faculty meetings in MAPS & Engineering. In SLMS Natasa has been meeting with meetings with all faculty tutors and now divisions. She reported the E-learning needs in SLMS  are also related to collaborative tools (for content delivery and learning activities), media rich interactive resources and ‘beyond the baseline’ Moodle activities.

We asked attendees if the Champions’ role should be more ‘formally ‘described but the audience was split. As one participant said, it was the informality of the group that made it successful.

We believe UCL is at the forefront of recognising that the complexity of e-learning provision in a modern university requires the development of a digitally literate community comprising a wide range of colleagues. We therefore discussed the personal development of Champions and Rosalind Duhs from CALT explained how the UCL Arena Fellowship programme could be an excellent route. Being a Champion and involvement in local projects would provide a sound basis for the development of portfolio case studies.

The session was concluded with an engaging presentation from Dr Adrien Desjardins who explained his role as a Champion in Medical Physics and Bioengineering. This was followed by further contributions from the audience and a lively discussion. We are planning a more formal review of the programme and this was a great start. Many thanks to all who contributed.

New UCL Moodle baseline

moodleMThe UCL Moodle Baseline that was approved by Academic Committee in June 2009, has now been updated after wide consultation on best current UCL practice.  The aim of the Baseline is to provide guidelines for staff to follow when developing Moodle courses in order for UCL students to have a consistently good e-learning experience. They are intended to be advisory rather than prescriptive or restrictive. These recommendations may be covered within a combination of module, programme and departmental courses.

Changes include the addition of a course usage statement explaining how students are expected to use their Moodle course. A communications statement is also now a requirement, in order to explain to students how they are expected to communicate with staff, and how often they can expect staff to respond. It is now a recommendation for staff to add (and encourage their students to add) a profile photograph or unique image, to make it easier to identify contributors in forums and other learning activities.

New guidelines for including assessment detail and Turnitin guidance have been added for those who use these technologies.

See the new UCL Moodle Baseline v2

Find out more about this and other e-learning news in the monthly UCL E-Learning Champions’ Newsletter.

Engagement! A tale of two MOOCs

What is the real educational experience of MOOC students? Some people seem to take strong positions on MOOCs without actually having completed one, after just ‘dipping in’. I felt this was not quite enough to judge what MOOC learning is about, so back in August I signed up two MOOCs running almost concurrently. Both were on the Coursera platform and both  – coincidentally – from Weslyan University. Modernism and Postmodernism is 14 weeks long , is still running and Social Psychology at a sprightly – and more normal – six weeks finished recently. I had actually completed a Coursera MOOCat the beginning of the year but as it was on a familiar subject I considered that taking subjects I knew little about would give me a more ‘authentic’ learner experience.

I thought it was important to avoid ‘dip-in-ism’ so I committed to completing both, even paying $40 to go on the Signature Track on the first one. This means Coursera verifies my identity when I submit assignments, both by typing pattern and face recognition. To set up face recognition initially I held my passport in front of the laptop camera and it scanned my photo. For typing recognition a short phrase is tapped out; Coursera now knows what a dismal typist I am.

Both courses were based around an hour or so of weekly video lectures but despite being out of the same stable, they turned out to be very different in design.

Modernism and Postmodernism was/is perhaps most ‘conventional’. Each week there were four to six short video lectures and a couple of original texts as assigned readings. That was it. The videos featured Weslyan president and star lecturer Prof Michael Roth. Most were professionally shot, though sometimes interspersed with lecture capture type clips from some of his classes. What was unexpected here was the quality of the video – although nice – was largely immaterial. The power and engagement was simply in Prof Roth’s remarkable narrative, essentially the story of modern Western thought since the Enlightenment and expressed in the works of Kant, Rousseau, Marx, Darwin, Flaubert, Baudelaire, Woolf and so on, not really a ‘grand narrative’ but a compelling intellectual bricolage. I was genuinely gripped by the story Roth was telling, and sometimes just read the transcripts (much quicker) when I was too busy to watch the video. The eight assessments, 800 word essays, were peer-marked and the twenty or so assignments I have looked at so far in the course are of quite a high academic standard. The peer marking approach is astonishingly valuable, by the way, as the other students usually present the material in a very different way; challenging and reviewing my understanding of that part of the course.

Social Psychology used video differently. The video of the lecturer in was slightly more ‘amateurish’ but the editing was far more sophisticated. Great effort had been taken to get permission to show and edit in some remarkable clips of experiments (including the infamous Stanford Prison Experiment), TED talks, interviews with psychologists and some public broadcasting documentaries. This was supported by chapter-length PDF extracts from major textbooks and reprints of papers. Together this was an astonishingly rich learning resource, the best I have seen on any online course, including many paid-for ones. Like the other course, the tutor voice of Prof Scott Plous was very clear and engaging but his written assignments were more diverse; reactions to an online survey, analysis of a web site and the ‘day of compassion’. The assignments – also peer-marked – were less good than the Modernism course but improved as the ‘drop-ins’ dropped out. The final assignments I read on compassion, from students in India, the Philippines and so on were genuinely moving. The idea that MOOCs encourage a superficial form of learning is misplaced, at least in this case. Participants had evidently reflected, sometimes quite deeply, on the sometimes challenging material.

Engagement and interaction In neither course did I especially follow the discussion threads, they were too fragmented. Social Psychology for example had 200,000 enrolments, 7000 forum posts in the first week and about 8000 students still active at the end. How can you have a ‘conversation’ in that environment? It made me wonder if ‘interaction’ our much-vaunted goal of many online courses is slightly overrated. Much more motivating to me as student was the strength of the narrative, the storyline, a bit like reading a good book in fact. Video proved an excellent way of getting that narrative across and the assignments in both made sure I assimilated at least some of the content and provided an important time frame to ensure I ‘kept up’. This ‘interaction light’ approach seemed to be in contrast with the Open University courses I have done, and indeed tutor on. These are deliberately designed around a series of regular interactions with fellow students and tutors and, being written by a teaching team, have a far less imposing narrative personality. Maybe in the MOOC environment, where ‘classical’ online interaction is necessarily weaker, design may necessarily focus not simply on interaction but engagement and that strong personal narrative may often be a key element. Just ‘dipping into’ a MOOC may completely miss this most important aspect.

 

Innovating pedagogy – 2013 trends report

Many of you will be familiar with the annual  Horizon reports, that describe emerging technologies likely to have an impact on learning and teaching. The current 2013 Higher Education report for example lists flipped classrooms, MOOCs, mobile apps and tablet computing as current areas of interest with augmented reality, game-based learning, ‘the internet of things’ and learning analytics as areas to watch out for in the near future. Matt Jenner reflected on the history of Horizons’ trend spotting on this blog earlier this year in When does a technology no longer become a technology?

However there is now a new kid on the trend-spotting block. The Open University this month published the 2013 version of Innovating Pedagogy, a series of reports that started last year and which “explores new forms of teaching, learning and assessment for an interactive world, to guide teachers and policy makers in productive innovation“. What makes this particularly interesting is that it has a UK focus and outlines some areas that are being actively discussed in UK universities, UCL included, but don’t feature (yet) in Horizons, though there is some overlap naturally.

The ten themes the OU pick this year are

  • MOOCs (of course)
  • Badges to accredit learning – an open framework for gaining recognition of skills and achievements
  • Learning analytics i.e. data-driven analysis of learning activities and environments
  • Seamless learning – connecting learning across settings, technologies and activities
  • Crowd learning – harnessing the local knowledge of many people
  • Digital scholarship – scholarly practice through networked technologies
  • Geo-learning – learning in and about locations
  • Learning from gaming –exploiting the power of digital games for learning
  • Maker culture – learning by making
  • Citizen inquiry – fusing inquiry-based learning and citizen activism

A fascinating and sometimes surprising list, and the report gives a quick overview of why the OU thinks these are or may be important and some links to further reading.

Just how good is your online course?

One of the perennial problems for both academic colleagues and learning technologists is trying to judge the educational value of online courses.Especially in blended learning the online ‘course’ is often just a component of a broader learner experience, and its role really can only be understood in the context of how it supports or extends ‘live’ activities. Thus what looks to a learning technologist like an unsophisticated ‘list of links’ in Moodle may actually support a rich classroom-led enquiry-based learning activity. It is hard to tell without speaking to the lecturer (or students) involved.

Nevertheless for modules which are wholly online or have a high use of technology a consensus has emerged as to what components are necessary to enable a ‘good’ course. One very practical example of this is the Blackboard Exemplary Course Program Rubric, which has gradually developed as a kind of sector standard since it was established in 2000, back then under the WebCT flag. The eight page rubric actually supports Blackboard’s Catalyst course competition (only open to Blackboard users, of course!) but the document can also be read as a platform-neutral checklist of good design, as applicable to Moodle as it is to Blackboard. Using the rubric course designers can evaluate how well their own course conforms to ‘best practices’ in four areas; Course Design, Interaction and Collaboration, Assessment and Learner Support. Each area is broken down into separate areas, with a checklist of ‘incomplete’ to ‘exemplary’ examples.

  • Course Design covers how clear the course goals and objectives are, the way the content is presented and any use of media, how learning design encourages students to be engaged in ‘higher order’ thinking and generally how the VLE is used to help student engagement.
  • Interaction and Collaboration includes communication strategies (an aspect so important we are considering including in the UCL Moodle baseline), how a sense of learner community is developed and ‘logistics’ i.e. quality and expectations of interaction.
  • Assessment is essentially about how assessment design aligns with the learning outcomes, the expectations on students and any opportunities for self assessment.
  • Learner support highlights the importance of orientation to the course and the VLE, clarity around the instructor role, links to institutional policies, accessibility and the role of feedback.

In short this is really a very useful checklist for people already running or currently designing programmes with a high online component and well worth a look. Using a checklist does not guarantee an ‘exemplary’ student experience but is simply a way to ensure that what are nowadays commonly regarded as critical components of success are fully considered in the course design and planning. Some of the sections may need some ‘interpretation’ or localisation and that is hopefully where E-Learning Environments can help!