Transforming exams: How IT works for BYOD e-exams
Concise paper
Mathew Hillier
Monash University
@mathewhillier
Andrew Fluck
University of Tasmania
@AndrewFluck
Catch this session
Monday 4 December, 2.10pm - 2.30pm
Stream 4
Room L209
Abstract
This paper focuses on the 'IT' side of a bring-your-own-device (BYOD) based e-Exam system developed as part of an Australian government funded project (transformingexams.com). The context for the use of our e-Exam solution is the supervised space of the exam room. A key element of the work is to enable authentic forms of assessment. To this end we have designed the solution to allow complex constructed problems that can be addressed using a range of contemporary 'e-tools of the trade'. The system works with BYOD where students boot their own laptop using a specially crafted USB stick that contains a standardised operating system and a suite of applications. By giving teachers and students access to contemporary software tools we are providing the opportunity to greatly expand the pedagogical landscape of the exam room encouraging more authentic assessment practices.
The paper provides technical details of both the e-Exam USBs used for the student test environment and newly developed tools used to streamline the deployment of exam data and retrieval of student responses. A phased strategy is outlined for moving from paper based exams via paper-equivalent e-exams through to post-paper exams involving multimedia, a range of software applications, quiz engine and then onto whitelisted network resources and fully logged Internet access enabled during an exam. The roles of groups (Us-s) and individuals (Me-s) within the process of running an e-exam are outlined by comparing current paper-based exam workflows and that of e-exams to provide a richer description of the approach.
About the authors
Mathew Hillier
Dr Mathew Hillier is a Senior Lecturer in the Office of Learning and Teaching at Monash University. Mathew is one of two co-leaders of the ASCILITE SIG for 'e-Assessment' and in this capacity is a co-host of the Transforming Assessment webinar series along with Prof Geoffrey Crisp.
He specialises in e-assessment and e-exams and teaches into the academic staff development program at Monash University leading the 'technology and space' theme. He has previously taught into Business, Information systems, Engineering and Arts programs at several universities in Australia, Singapore and Hong Kong. Mathew is currently the leader of the 'Transforming Exams' project developing a toolset for authentic, computerised, high-stakes assessment (e-Exams). The project covers 10 university partners and is funded by a half-million dollar Australian government grant. He is also leading the development of a multi-language computerised exam platform for use in national professional translator accreditation. More about Mathew at http://ta.vu/mathewhillier
Andrew Fluck
Dr Andrew Fluck is a Senior Lecturer in Information Technology in the Faculty of Education at University of Tasmania.
Dr. Fluck trained as a teacher in Bristol, England. He has taught science, mathematics and computing in Nigeria, England and Australia. His numerous publications reflect his research interests in the transformational potential of computers in education. He was a co-author of the government report 'Making Better Connections' and the book 'Seven steps to ICT integration'. His funded research investigates the use of computers to teach integral calculus and quantum mechanics in primary schools; and eExaminations, where students take their own computers into the exam hall. He is a past secretary of the Australian Council for Computers in Education (ACCE) and is now the chair of Working Group 3.3 (research into educational applications of information technologies) for IFIP/UNESCO. Andrew is also an avid longbow archer and continental archery judge. Andrew's website http://Andrew.Fluck.id.au
Understanding students’ views on feedback to inform the development of technology-supported feedback systems
Concise paper
Linda Corrin
University of Melbourne
@lindacorrin
Paula de Barba
University of Melbourne
@paula_barba
Catch this session
Monday 4 December, 2.10pm - 2.30pm
Stream 3
Room L206
Abstract
In an increasingly expanding higher education system, students have routinely said that they don’t get enough access to feedback to support their learning. While this feedback loop is recognised as a critical issue, the growing use of technology as part of teaching and learning could provide some solutions to this problem. The emergence of the field of learning analytics has the potential to provide mechanisms for reducing some of the concerns students have about receiving feedback. However, a greater understanding of how learning analytics can be used to provide meaningful assessment feedback to students is needed.
This paper presents the initial findings from a study that investigated students’ preferences for the delivery of assessment feedback to improve their learning. The findings show that there is a diversity of student perspectives on what feedback is most useful for their learning which is influenced by the type of assessment, the discipline in which the assessment takes place, the year level of the student and the ability to compare performance to others. The outcomes of this study provide evidence of what students want when it comes to analytics-based feedback which can be used to inform the development of guidelines for how such feedback can be designed and delivered in higher education.
About the authors
Linda Corrin
Dr Linda Corrin is a Senior Lecturer in Higher Education in the Williams Centre for Learning Advancement, Faculty of Business and Economics, at the University of Melbourne. In her current role, she provides support for curriculum development, delivery and assessment to staff in the faculty. Her research interests include students' engagement with technology, learning analytics, feedback, and learning design. Currently, she is working on several large research projects exploring how learning analytics can be used to provide meaningful and timely feedback to academics and students. Linda is member of the University of Melbourne’s Learning Analytics Research Group and co-founder of the Victorian and Tasmanian Learning Analytics Network. She is also a co-coordinator of the ASCILITE Learning Analytics Special Interest Group.
Paula de Barba
Paula de Barba is a Research Fellow in Higher Education with the Melbourne Centre for the Study of Higher Education at The University of Melbourne. Her current research is in the areas of educational psychology and educational technology. Paula is interested on how students' cognition and emotions influence their learning, and how technology can support and promote learning. Constructs investigated include self-regulated learning, achievement motivation, interest, and feedback.
Dramaturgy: A sociological perspective for conceptualising Me. Us. IT in the context of online learning
Experimental session
Dawn Gilmore
Swinburne University of Technology, Faculty of Health, Arts and Design
@dawngilmore9
Catch this session
Monday 4 December, 3.30pm - 4.30pm
Stream 1
Room H102
Abstract
A dramaturgical analysis seeks to understand people’s everyday lives through the aspects of a theatre. I propose that this is a useful analogy for illustrating how students experience learning in an online subject. Based on this analogy, an online student is an actor who crafts performances within the front stage and backstage of a particular university subject.
At the start of this session attendees are introduced to three theatre stages. These stages are defined as the front stage (the LMS), the backstage online (internet websites and social media), and the backstage offline (conversations with family, friends, and colleagues). Following this, attendees will meet student-avatars who play the theatre roles of performer, cameo, extra, and stagehand. The creation of the student-avatars was informed by a fourteen-month study of online university students. During this time data was collected from 224 student observations, 120 questionnaires, 26 interviews, and the content analysis of 1,857 discussion board posts.
Each student-avatar will communicate their learning journey by briefly sharing how they experience learning in the front stage, backstage online, and backstage offline. Through their stories, their preferences and patterns for individual experiences, group experiences, and the tools they most commonly use for university related tasks come to light. With the student-avatars in mind, attendees will design a short learning experience that acknowledges how students enact multiple identities across the three performance stages. The learning experiences will be collated into an online book that will be shared with conference attendees. This book symbolises how we can harness our collective intelligence to support online students.
About the authors
Dawn Gilmore
Dawn Gilmore has over 15 years of practice and research in teaching and learning in higher education. She has worked in institutions in the United States, Australia, Japan, England, China, and most recently South Africa. She has a M.S.Ed. in Intercultural Communication from the University of Pennsylvania and a B.S.Ed. in Social Sciences from Temple University. Her Ph.D. research explored where and with whom university students experience learning. During her doctoral candidature she was a Visiting Scholar at in the Centre for Teaching, Learning, and Development at the University of the Witwatersrand (South Africa), where she researched the role of communities of practices in a Post Graduate Diploma of Higher Education. Overall, her research seeks to understand how students’ learning is situated across time and spaces. She also has broader interests in learning, community, and technology. Dawn is currently writing a book that is largely based on the content from this presentation.
Future happens: Hack your way to influencing and changing pedagogical and technological strategy and practice
Experimental session
Peter Bryant
London School of Economics and Political Science
@peterbryantHE
Catch this session
Monday 4 December, 3.30pm - 4.30pm
Stream 4
Room L209
Abstract
Using the changehack approach successfully run in the UK for the last two years by Future Happens (http://www.futurehappens.org - a collaboration between two leading UK institutions, the London School of Economics and the University of the Arts, London), this experimental session is designed to collectively engage participants in changing the discourses around the role of technology in shaping institutional/faculty wide pedagogical change. This lightning changehack will generate approaches to scaling and sustaining the lessons and innovations that arise from grassroots practice into approaches that can be included in strategic thinking across disciplines, levels, cohorts and potentially across the whole institution.
This workshop will challenge you to think about the ways you are able to influence your institutions strategic direction and commitments to technology and learning and be a part of the conversation that shapes how they do it. Attendees will participate in a collective hack that draws on the power of the crowd to solve problems. Previous Future Happens hacks in the UK have collectively generated insightful, useful and pragmatic ways to bridge the discourses between the practices of learning technology and how they can be scaled up to be part of the institutional, faculty or School wide strategic approach to innovative pedagogy. Attendees will collectively own the outputs which will be shared globally as part of the Future Happens movement.
About the authors
Peter Bryant
Peter Bryant is the Head of Learning Technology and Innovation at the London School of Economics and Political Science, UK. He leads programmes and initiatives to transform the educational experience at the LSE through the innovative use of technology and digital pedagogies. His team recently was the Overall Gold Award for Innovative Pedagogy at the Wharton-QS Stars Reimagine Education awards. He was previously a Principal Lecturer in Educational Technology and Development at the University of Greenwich. Peter has over twenty years’ experience as a lecturer, Head of Department and curriculum designer, working into two countries (in HE and VET). He is an active researcher in both educational technology and pedagogy. Peter is the co-founder of the Future Happens initiative which uses innovative approaches to problem-solving and change management to engage the wider sector in debates around technology, pedagogy and the future of the University.
Challenges and tensions in the role of the LMS for medical education: Time for the "next generation LMS"?
Full paper
Jill Lyall
Australian National University
@JillACTS
Katharina Freund
Australian National University
@katiedigc
Alexandra Webb
Australian National University
Catch this session
Monday 4 December, 11.30am - 12pm
Stream 1
Room H102 Allison Dickson Lecture Theatre
Abstract
In the context of discussions of a “next generation LMS” and other contemporary challenges in higher education, this case study looks at the iterative process a team of educational designers and Medical School academics used in a review of the Medical School LMS sites. Adopting the framework of the actor network theory, this reflective process discovered the tensions, dynamics and issues involved, and worked to gain and maintain key Medical School staff engagement and support for the review and for any changes that might be recommended. This paper reflects on emerging possible models for technology-enhanced learning beyond our current institutional LMS while acknowledging the institutional constraints on learning innovation within the global higher education context. Next generation LMS models may provide a more flexible future solution that could be applicable not just to medical education, but to higher education generally.
About the authors
Jill Lyall
Jill has a background in social sciences, community services, and adult education. Jill has worked for many years in Vocational Education and Training in a TAFE environment in Western Australia. During that time she developed an interest in technology for learning, and developed her skills for online learning. Since early 2015, Jill has been working on a range of projects with ANU Online, helping to create online materials for post-graduate courses and creating support and training resources in technology enhanced learning for academics at ANU.
Katharina Freund
Katie advises academics on eLearning design and initiatives, researches innovative solutions in education technology, creates digital media resources, and trains staff on Wattle (Moodle) learning management system and other digital tools for teaching and learning. She is also a researcher in education technology and digital communication.
Alexandra Webb
Dr Webb has more than 20 years experience teaching anatomy into undergraduate & postgraduate medicine, allied health & science programs in Australia & the United Kingdom. She has extensive proficiencies in leading the development & implementation of new curricula & resources. She is an innovative educator & educational researcher who takes measured risks in trialling the effectiveness of novel teaching & learning approaches such as serious games, touchscreen technology, adaptive learning, virtual patients & eBooks. In her current role, she is leading the development & implementation of technology enhanced learning & teaching for the School’s MChD as well as creating a flexible collaborative Professional Development program to build capacity in teaching & learning. Dr Webb’s educational practice has been recognised with multiple awards, including Senior Fellowship of the Higher Education Academy & a national Australian Award for University Teaching.
Playing the education system: Competing, exploring, socialising, distrupting, but always engaging
Experimental session
Dan Laurence
La Trobe University
@D4n_
Catch this session
Monday 4 December, 3.30pm - 4.30pm
Stream 6
Room T120
Abstract
Through discovering your ‘gamer type’ as part of this experimental session participants will actively explore, compete, socialise and disrupt their way into the enquiry: what aspects of the education system are ‘gameful’, and how do different students play?
There is contention around the usefulness of the idea that learners fall into different types. Some research shows that curating teaching based on ‘learning styles’ (sensory processing) is of questionable value. However, evidence from learning analytics shows that there are radically differently behaviours exhibited by students and there are impassioned calls for increased personalisation in education.
After a very brief introduction to some leading theories on student engagement we quickly segue into a ‘Gamer Type’ quiz that will determine participants teams. The teams will then compete live using a series of leading game/learning apps as we delve deeper into the enquiry of what aspects of education are gameful, are there different ways students play and if so, does it matter?
BYO device or phone.
About the authors
Dan Laurence
Dan Laurence is a senior educational designer and has taught Graduate Certificate students on the subject of games in education. In 2015 Dan won the Vice Chancellor’s Teaching Excellence award for his work designing and employing game principles at Swinburne University (and accounting for their impact). The following journal article [ http://rdcu.be/uOJG ] has been published detailing the implementation. Dan has a long prior history working in interactive media and has produced games that are used in in around 40 universities across the world.
A framework for program wide curriculum transformation
Full paper
Angela Nicolettou
RMIT University
@ANicolettou
Andrea Chester
RMIT University
Spiros Soulis
RMIT University
@SSoulis
Catch this session
Monday 4 December, 11.30am - 12pm
Stream 2
Room R113
Abstract
Designing and delivering higher education programs in a global climate of constant change, technological advances and uncertain futures leads to the need for curriculum transformation practices that are innovative and responsive.
This paper describes a university-wide approach to developing a framework for program level transformation that is strengths-based, data-informed and design-led. A strengths-based approach builds on good practice, creating a space that is positive and forward looking. Data-informed practice and the inclusion of data wranglers on the project allowed for conversations about the known, unknowns and desirable directions to take place and inform directions. Design-led practices introduced design thinking principles such a building empathy and co-design with students, alumni and industry. The emergent framework has three key stages: vision, design and build. The vision stage focuses on the program team, its students, industry and desired direction for transformation. The design stage focuses on defining challenges, ideating, co-designing and creating a plan for development. The build stage uses a rapid prototyping and iterative approach to development that incorporates user testing early in the stage.
The project has delivered a framework for program level transformation and innovation and has shown that a strengths-based approach that is data informed and engages with students as co-designers has the capacity to unite teams, inform program visions and allow for innovative practices to emerge. Taking a learner experience approach to design also highlighted the value in engaging students and industry in curriculum design from the start of the process rather than simply as end users.
About the authors
Angela Nicolettou
Angela Nicolettou is the Manager of the Digital Learning Team in the College of Design and Social Context at RMIT University. Angela has over 20 years experience in Higher Education, leading numerous curriculum design teams, teaching, program management of Graduate Certificates in tertiary teaching, academic development and educational design. A focus of Angela’s work has been teaching with educational technologies, curriculum design, assessment and sessional teacher training.
Andrea Chester
Andrea is the Head of the School of Education and Deputy Pro Vice-Chancellor, Learning and Teaching at RMIT University. She has a background in psychology education and practice, with more than 20 years experience as an academic. Over the years Andrea has taught across the psychology curriculum, with a particular interest in first year and transition issues and she continues to teach in both first and third year. Her research focuses on learning and teaching, with an emphasis on how social capital can enhance learning. Her work has explored how partnerships between students can be mutually beneficial, how partnerships between staff can enhance reflective practice and how partnerships between students and staff can be leveraged to transform learning.
Spiros Soulis
Spiros Soulis is a Learning Designer in the College of Design and Social Context at RMIT University. Spiros experience spans across ESL and higher education, in leading and managing countless learning design teams as well as high profile projects in learning and teaching. His main research areas are in Design Thinking, design for learning and learner experience.
A learning analytics view of students’ use of self-regulation strategies for essay writing
Full paper
Kelly Trezise
University of Melbourne
@Kelly_Trezise
Paula de Barba
University of Melbourne
@paula_barba
David Jennens
Vericus Pty Ltd
@cadmus_io
Alexander Zarebski
Vericus Pty Ltd
Robert Russo
Vericus Pty Ltd
@cadmus_io
Gregor Kennedy
University of Melbourne
Catch this session
Monday 4 December, 10.30am - 11am
Stream 5
Room C204
Abstract
Essay writing is a fundamental part of higher education. Students’ use of self-regulatory skills, such as time management and planning and writing strategies, while writing essays predicts better writing quality. Current characterisations of the relationship between self-regulation and essay writing are limited by the difficulty of assessing self-regulation in real-life essay writing contexts.
This paper reports on a novel approach to examine students’ use of self-regulation strategies in a real-life setting, using learning analytics. Four case studies are presented to illustrate similarities and differences in students’ use of time management and planning and writing strategies. Participants managed their time in very different ways to complete the assignment. They were active over a different number of days, engaged in sessions of different durations, and at different times of the day. The participants used variety of approaches to their writing: one participant started early and allowed editing time, another typed gradually over a number of days, and two participants waited until the due date to complete the essay, with varying amounts of editing. Findings from this research contribute to a novel detailed empirical evidence of different essay preparation behaviour in real-life settings. After further studies with a variety of essay types and student samples, there may be significant value in using the approach outlined in this paper as the basis of tools to provide students with advice and support in their essay preparation.
About the authors
Kelly Trezise
Dr Trezise is currently a Research Fellow at the Melbourne Centre for the Study of Higher Education, working with the Educational Technology Research Group. She also works at the Melbourne School of Psychological Sciences. She is primarily interested in identifying individual differences in patterns of learning. Dr Trezise researches how cognition, emotion, and educational contexts, and their effect on learning processes and educational outcomes. With the CHSE she examines how educational technology and cognition interact during learning. Kelly uses a variety of analytical methods, including finite mixture models, to examine processes that contribute to learning and development. In her PhD completed at the Melbourne School of Psychological Sciences, Kelly examined maths anxiety¬cognition interactions and their relationship with algebraic problem solving in adolescents.
Paula de Barba
Paula de Barba is a Research Fellow in Higher Education with the Melbourne Centre for the Study of Higher Education at The University of Melbourne. Her current research is in the areas of educational psychology and educational technology. Paula is interested on how students' cognition and emotions influence their learning, and how technology can support and promote learning. Constructs investigated include self-regulated learning, achievement motivation, interest, and feedback.
David Jennens
David holds a PhD in Particle Physics from The University of Melbourne on measuring the decay rate of the Higgs Boson into tau lepton pairs. He performed his research in collaboration with the ATLAS experiment at the Large Hadron Collider with the European Organisation for Nuclear Research (CERN) and the Center of Excellence for Particle Physics (COEPP). During this research period he contributed to the ATLAS detector calibrations. This work led to him being included on the ATLAS collaboration author list for several papers, including the discovery of the Higgs Boson. David currently leads a team using data collected through Cadmus to improve the quality of teaching and learning at universities.
Alexander Zarebski
I am an applied mathematician in the University of Melbourne's School of Mathematics and Statistics. After obtaining a Master's degree in mathematics and statistics in 2014, I turned my attention towards modelling infectious diseases, primarily influenza, to produce forecasts of transmission through the population. In addition to my work in quantitative epidemiology, I have a keen interest in education. This has led me to tutor a range of subjects in the School of Mathematics and Statistics. I also work as a data scientist at the technology start-up Vericus where I develop algorithms to analyse students' interactions with educational technology.
Robert Russo
Robert is the Chief Product Officer at Cadmus and is responsible for product strategy. Robert graduated in 2014 from the University of Melbourne with a Masters of Engineering. Under Robert, Cadmus’ focus has been on ensuring all product decisions align with best teaching and learning practice.
Gregor Kennedy
Gregor Kennedy is Pro Vice-Chancellor (Teaching and Learning) at the University of Melbourne and Director of the Melbourne Centre for the Study of Higher Education. Gregor is an international leader in educational technology research and development, particularly in the context of higher education. He has longstanding research interests in contemporary learning design and emerging technologies, educational technology research and evaluation, interactivity and engagement in digital learning, 3D immersive virtual environments, and the use of learning analytics in digital learning environments. He has published widely in these areas and is a regular keynote and invited presenter at local and international conferences. As Pro Vice-Chancellor (Teaching and Learning) Gregor leads the University's strategy in teaching, learning and assessment, curriculum innovation, and the use of learning technologies and learning analytics.
Blended learning as a disruption in a vocational education building program
Full paper
Meg Colasante
La Trobe University
@meg_colasante
Cathy Hall-van den Elsen
RMIT University
Catch this session
Monday 4 December, 11am - 11.30am
Stream 4
Room L209
Abstract
A building and construction teaching team in a vocational education school (within a university in Melbourne) introduced a blended learning model to one-third of their program. Traditionally, building students are accustomed to a high ratio of face-to-face learning, therefore, this new model disrupted the experience of both teacher and student. The model was examined using e-learning evaluation research methodology and findings are presented using the framework of Glazer’s (2012) characteristics of blended learning. Examination of the program identified areas in need of attention, such as active learning and online interaction and communication. Finally the authors promote the use of Glazer’s framework as a pedagogical evaluation tool for blended learning designs, while drawing out a particular focus on teacher presence as a distinct item in this framework.
About the authors
Meg Colasante
Meg Colasante works at La Trobe University as Lecturer, Educational Development (Digital Learning Strategy), currently within the La Trobe Online team within La Trobe Learning and Teaching. This follows two decades of teaching, educational design, and academic development roles at RMIT University. Meg is also a PhD candidate with Deakin University, studying university teacher practices with digital video.
Cathy Hall-van den Elsen
Dr Cathy Hall-van den Elsen is a retired academic who has worked in learning and teaching for a number of years. Until August 2014 Cathy was the Senior Manager, Academic Development Group in the College of Business at RMIT University. Cathy has contributed to projects associated with the scholarship of learning and teaching, teacher induction, transnational teaching, and innovative teaching practices, and has recently consulted on several projects at RMIT University.
Addressing inconsistency in use of the LMS: A collaborative approach
Full paper
Elizabeth Masterman
University of Oxford, UK
@acitoxford
Catch this session
Monday 4 December, 10.30am - 11am
Stream 1
Room H102
Abstract
Inconsistency in the use of the learning management system (LMS) by academic staff is a source of dissatisfaction among university students in the UK. One solution is to establish a set of minimum standards (or baseline) for LMS use within an individual institution. Another is to supply templates – frameworks for LMS course sites – with a view to providing students with a seamless experience in their interactions with the LMS.
This paper describes how the issue of inconsistency was addressed at a leading research university in the UK through an exploratory project, WebLearn Improved Student Experience (WISE). The widespread devolution of responsibility for site management to administrative staff, together with the ‘maverick’ creation of course sites by those academics who chose to engage with the WebLearn LMS, had resulted in unevenness in students’ access to learning materials. The project team engaged in close collaboration with 19 departments in order to achieve the immediate purpose of improving uptake of, and consistency in, their LMS presence. The ultimate aim was to develop a support package comprising LMS templates and ‘best practice’ guidelines that would enable departments in the future to achieve the same objective, either unsupported or with minimal assistance from the central team of learning technologists. The project was evaluated using a modification of the Innovation Histories method. The evaluation findings additionally threw into relief the complex social and cultural factors at play that can inhibit a consistent student experience in an institutional LMS.
About the authors
Elizabeth Masterman
Liz Masterman holds a PhD in Educational Technology from the University of Birmingham, UK, and is a senior researcher in the Academic IT Services group at the University of Oxford. Her research focused initially on Learning Design; projects included an evaluation of LAMS, and the Phoebe Pedagogy Planner and Learning Designer tools. Liz has also conducted research into the student digital experience, OER and open educational practice. She is currently involved in three institutional projects: a review of the centrally supported LMS, a trial of electronic essay exams, and the design and implementation of a Learning Design model for Oxford. Other activities include editing the Academic IT Services blog and co-ordinating the annual OxTALENT competition, which rewards the creative use of technology in teaching, learning, research and outreach at the University. In 2015 Liz co-chaired the annual conference of the UK’s Association for Learning Technology.