THE IDEAL ASSESSMENT TOOL: VALID AND SECURE E-ASSESSMENTS WITH MOODLE AND SAFE EXAM BROWSER MOODLE MOOT, 2013.09.06 Tobias Halbherr, tobias.halbherr@let.ethz.ch Daniel Schneider, daniel.schneider@let.ethz.ch April 2013 LET Lehrentwicklung und -technologie 1
INTRODUCTION Dienstag, 10. September 2013 LET Lehrentwicklung und -technologie 2
Terminology e-assessment online-assessment (online-exam) disconnected computer internet intranet LET Lehrentwicklung und -technologie
Online-Exams at ETH Zurich online offline summative formative centralized decentralized LET Lehrentwicklung und -technologie
Why do Exams Matter? «Will this be on the exam?» Exams strongly influence what and how students learn. motivation / focus / direction / perception «Will I pass the exam?» Exams strongly influence which students complete a curriculum. skills / interests / strengths / weaknesses «What should I test in a exam?» Exams subtly yet fundamentally influence the future direction of a discipline. Tuesday, September 10, 2013 Course Title 5
ADVANTAGES OF ONLINE-EXAMS Dienstag, 10. September 2013 LET Lehrentwicklung und -technologie 6
Criteria for Measurement Quality validity reliability (objectivity) LET Lehrentwicklung und -technologie 7
Examinations as Measurements competency grading learning objective marking examination tasks exam LET Lehrentwicklung und -technologie 8
Classification of Question Types closed SC MC matching ordering drag & drop semi-closed short answer text numerical hotspot open essay 3rd party SW sketch LET Lehrentwicklung und -technologie 9
Advantage: Legibility, Editing LET Lehrentwicklung und -technologie 10
Alignment Learning Objective describes Competency measures Learning Activity enables acquisition of Assessment Task 11
Advantage: Alignment learning activity assessment task target competency LET Lehrentwicklung und -technologie 12
Quality Efficiency LET Lehrentwicklung und -technologie Online-Exams: Advantages legibility editing SW-based competencies multimedia interactivity feedback quality control marking automatic manual administration processes collaboration question pools LET Lehrentwicklung und -technologie 13
INFRASTRUCTURE FOR ONLINE-EXAMS Dienstag, 10. September 2013 LET Lehrentwicklung und -technologie 14
LET Lehrentwicklung und und -technologie -technologie Questions to ask in the beginning What kind of examinations? > Computer-based exams > Online examinations > Centralized exams Which infrastructure to use for your examinations? > Special assessment computer rooms? > Computer lecture rooms? > Student computers? How can you prevent manipulation, fraud, plagiarism and the obtaining of information from the Internet? 15
One possible answer: Dienstag, 10. September 2013 LET Lehrentwicklung und -technologie 16
LET Lehrentwicklung und und -technologie -technologie What is the Safe Exam Browser SEB? Secured web browser application for conducting online examinations using learning management systems and web-based quiz/e-assessment systems available for free, open source 17
LET Lehrentwicklung und und -technologie -technologie What is Safe Exam Browser SEB doing? SEB prevents opening any website switching to undesired applications operation system specific key combinations and functions quitting the secure browser during the exam 18
LET Lehrentwicklung und und -technologie -technologie What is Safe Exam Browser SEB doing? SEB allows connecting to web based learning management systems and web-based quiz/e-assessment systems use of selected third party applications during the examination 19
Architecture Kiosk Application starts Browser URL to Test Learning Management Systems Moodle, ILIAS,... starts Third Party Application Dienstag, 10. September 2013 LET Lehrentwicklung und -technologie 20
LET Lehrentwicklung und und -technologie -technologie The SEB-Browser Mac OS X Windows WebKit Mozilla Gecko / XULRunner (Firefox) 21
LET Lehrentwicklung und und -technologie -technologie SEB s advantages Application! 22
LET Lehrentwicklung und und -technologie -technologie SEB s advantages: Application Easy to install No complex administration of exam computers necessary No special network configuration necessary in the exam room 23
LET Lehrentwicklung und und -technologie -technologie SEB s advantages: Application vs. booting special OS Don t have to create, test and regularly update special boot images Very good compatibility with a wide range of hardware Windows and Mac OS X third party applications can be used in parallel with the exam 24
Workstations Dienstag, 10. September 2013 LET Lehrentwicklung und -technologie 25
SUMMARY Dienstag, 10. September 2013 LET Lehrentwicklung und -technologie 26
When do Online-Exams provide benefits? essay type MC & short-answer SW based competencies multimedia multiple examiners large groups of candidates interactivity open book time-critical LET Lehrentwicklung und -technologie 27
Q&A Tobias Halbherr LET Lehrentwicklung und technologie tobias.halbherr@let.ethz.ch LET Lehrentwicklung und -technologie
ONLINE-EXAMS: ORGANIZATIONAL AND TECHNOLOGICAL CHALLENGES IN TRANSITIONING FROM PAPER TO ONLINE MOODLE MOOT, 2013.09.06 Tobias Halbherr, tobias.halbherr@let.ethz.ch Daniel Schneider, daniel.schneider@let.ethz.ch April 2013 LET Lehrentwicklung und -technologie 29
Online-Exmams: Where Challenges & Opportunities Challenges Stakeholders Infrastructure Processes Opportunities Improved quality Improved efficiency Increased design possibilities Dienstag, 10. September 2013 LET Lehrentwicklung und -technologie 30
Question Types: Costs & Benefits quality efficiency IT-resources MC = short answer = interactive/mm ( ) essay 3rd party SW ( ) sketch - - - formula - - - LET Lehrentwicklung und -technologie 31
Exam-Scenarios browser based design delivery tech-support IT-resources one-to-one browser based open closed browser based multimedia online & paper 3rd party SW CAD tablet???? LET Lehrentwicklung und -technologie 32
Designing Exams: What to consider WHO do I wish to assess? WHAT do I wish to assess? HOW do I wish to assess? SCENARIO LET Lehrentwicklung und -technologie
Service & Support for Online-Exams: What needs to be considered? method, psychometrics, alignment regulations LMS exam delivery & logistics service & support secure exam environment roominfrastructure IT-infrastructure Wednesday, January 30th, 2013 LET Lehrentwicklung und -technologie 34
Service and Support for Online-Exams: Who is involved? examiners faculty development legal department service & support e-learning center IT-services roominfrastructure examinations office Wednesday, January 30th, 2013 LET Lehrentwicklung und -technologie 35
Q&A Tobias Halbherr LET Lehrentwicklung und technologie tobias.halbherr@let.ethz.ch LET Lehrentwicklung und -technologie