eas Online Conference

Transkript

1 eas Online Conference 18 September 2014, 07:00 GMT Safe Exam Browser: A modular approach to secure and flexible online-exams Tobias Halbherr & Daniel Schneider Educational Development and Technology (LET), ETH Zürich Your hosts Professor Geoff Crisp, Dean Learning and Teaching, RMIT University geoffrey.crisp[@]rmit.edu.au Dr Mathew Hillier, Institute for Teaching and Learning Innovation, University of Queensland mathew.hillier[@]uq.edu.au Transforming Assessment is supported by Just%to%let%you%know:%% By#par'cipa'ng#in#the#session#you# acknowledge#and#agree#that: The#session#may#be#recorded,#including# voice#and#text#chat#communica'ons#(a# recording#indicator#is#shown#inside#the# webinar#room#when#this#is#the#case).#we# may#release#recordings#freely#to#the#public# which#become#part#of#the#public#record.# We#may#use#session#recordings#for#quality# improvement,#or#as#part#of#further#research# and#publica'ons.

2 Safe Exam Browser: A modular approach to secure and flexible online-exams Tobias Halbherr, Daniel Schneider, & Thomas Piendl, Educational Development and Technology (LET), ETH Zürich Tobias Halbherr et al. 6/10/2014 2

3 Introduction Tobias Halbherr et al. 6/10/2014 3

4 «Online Exams»: Terminology! online! summative! with credit! invigilated e-assessment online-assessment (online-exam) disconnected computer internet intranet Tobias Halbherr et al. 6/10/2014 4

5 Online Exams at ETH Zurich Swiss Federal Institute of Technology! 2007 first pilots, 2010 regular service! ~45 exams with ~5000 students in spring semester 2014! Large percentage of selective, high-stakes examinations! Quality oriented service strategy Tobias Halbherr et al. 6/10/2014 5

6 Where do we want to go? Tobias Halbherr 6/10/2014 et 6 al.

7 What is the central question in the life of a student? Will this be on the exam? Tobias Halbherr et al. 6/10/2014 7

8 What should a «good» exam achieve?! Assess competences! valid! reliable! fair! Support student learning! Single most influential factor on student learning behavior!! What, when & how students learn Tobias Halbherr et al. 6/10/2014 8

9 Goals 1. Improve learning outcomes by making examinations more meaningful and motivating to students 2. Improve measurement quality of competence assessments 3. Improve efficiency of preparing, conducting, and scoring examinations 4. Create additional points of contact with examiners for pedagogical support 9

10 Requirements: Technology! Secure! Reliable, robust! Easy to use, familiar (for students)! Easy to customize and manage (for administrators & lecturers) Tobias Halbherr et al. 6/10/

11 How do we get there? Tobias Halbherr 6/10/2014 et 11 al.

12 What is the Safe Exam Browser SEB?! Secure lockdown application with integrated web browser! Runs on exam clients (Windows, Mac OS X, in future ipad)! Connects to learning management systems and other web-based online examination systems! Controls access to third party applications 12 Thursday, April 18, 2013 Educational Development and Technology (LET)

13 What is SEB?! Publicly funded! Open source! Freeware! Global user base 13 Thursday, April 18, 2013 Educational Development and Technology (LET)

14 What is SEB?! Modular, highly configurable system! Can easily be used on computers managed by the university and on BYOD student notebooks! Uses encrypted and signed individual configuration for each exam! Plugins for LMS like Moodle and ILIAS allow to ensure integrity of the SEB application and its settings used for an exam 14 Thursday, April 18, 2013 Educational Development and Technology (LET)

15 Architecture Kiosk Application starts Browser URL to Test Learning Management Systems Moodle, ILIAS,... starts Third Party Application 15 Thursday, April 18, 2013 Educational Development and Technology (LET)

16 How to use SEB with BYOD! Send students a download link for SEB and your.seb configuration file! Students download and install SEB! Students double click on the.seb file, SEB starts and gets configured to start the exam or to display a exam portal page! An exam portal page lists current exams, the links can contain another.seb file with individual configurations for each exam 16 Thursday, April 18, 2013 Educational Development and Technology (LET)

17 Example: How to set up SEB for a Moodle quiz! Download and install SEB! Generate a.seb configuration file for your exam! Set Moodle quiz option Secure Browser: Safe Exam Browser! Optional: Use the Moodle quiz access rule plugin, copy-paste the browser exam key generated by SEB for your exam configuration to the quiz settings form! Optional: Place a quit link onto your quiz feedback page 17 Thursday, April 18, 2013 Educational Development and Technology (LET)

18 Prüfungssetup mit lokalen Rechnern und Safe Exam Browser (SEB) Abgesicherter Client. Kioskkomponente Browser komponente l Verifikation Zugang LMS in Prüfungskonfig.. Prüfungsmodul.. A1 A2 A3 Prüfungsclient LMS Server Tobias Halbherr et al. 6/10/

19 How to perform an exam with SEB! After started, SEB locks down the computer! The exam is started in the SEB browser! Students cannot open other websites! Students cannot switch to other applications, unless you configured allowed third party applications to be used during the exam! After the exam is submitted, students can click the quit link on the quiz feedback page to quit SEB or you can communicate a quit password when the exam time is over 19 Thursday, April 18, 2013 Educational Development and Technology (LET)

20 Taking Exams to a new Level Tobias Halbherr 6/10/2014 et al.

21 Exam Tasks Today learning exam application / competence Tobias Halbherr et al. 6/10/

22 Example learning exam target competence Tobias Halbherr et al. 6/10/

23 The Goal: Authentic, Competence Oriented Exam Tasks learning exam application / competence Tobias Halbherr et al. 6/10/

24 Three Types of Functions of Computer in Online-Exams exam engine tool simulation Tobias Halbherr et al. 6/10/

25 Prüfungssetup mit Virtueller Desktop Infrastruktur, SEB und Drittapplikationen Abgesicherter physischer Client.. Abgesichert & in Prüfungskonfig. LMS in Prüfungskonfig. Kioskkomponente Kioskkomponente Prüfungsmodul.. VDI Client Zugang Browser komponente l Verifikation Zugang A1 A2 A3 Web App1 App2 Dateien ABC Web App1 schreiben Dateien lesen XYZ Hochladen Physischer Prüfungsclient Virtueller Desktop Virtuelle Desktop Infrastruktur LMS Server Tobias Halbherr et al. 6/10/

26 Prüfungssetup mit Linux-Rechnern Client in Prüfungskonfig.. Nur Prüfern zugänglich. App1 schreiben lesen Dateien XYZ kopieren Web App2 App3 Dateien ABC Prüfungsclient Dateiserver Tobias Halbherr et al. 6/10/

27 «Werkzeuge» für die Gestaltung authentischer, kompetenzorientierter Online-Prüfungen Prüfung Tobias Halbherr et al. 6/10/

28 Thank you for your attention Q&A Tobias Halbherr Daniel Schneider Thomas Piendl

29 Criteria for Measurement Quality validity reliability (objectivity) 29

30 Examinations as Measurements competency grading learning objective marking examination tasks exam 30

31 Classification of Question Types closed SC MC matching ordering drag & drop semi-closed short answer text numerical hotspot open essay 3rd party SW sketch 31 LET Lehrentwicklung und -t

32 Advantage: Legibility, Editing 32

33 Advantage: Alignment learning activity assessment task target competency 33

34 Online-Exams: Advantages Quality legibility editing SW-based competencies multimedia interactivity feedback quality control Efficiency marking automatic manual administration processes collaboration question pools 34

35 Examination Rooms 35

36 Workstations Monday, 6 October 1436

37 Workstations for Online-Exams! ergonomics! orderly & clearly arranged! lighting! privacy! noise! cheating! proctoring! visibility / ease of inspection! wardrobe! operation! exam / regular! reservation! preparations & postprocessing Monday, 6 October 1437

38 When do Online-Exams provide benefits?! essay type! MC & short-answer! SW based competencies! multimedia! multiple examiners! large groups of candidates! interactivity! open book! time-critical 38

39 Online-Exmams: Where Challenges & Opportunities Challenges! Stakeholders! Infrastructure! Processes Opportunities! Improved quality! Improved efficiency! Increased design possibilities Monday, 6 October 1439

40 Question Types: Costs & Benefits quality efficiency IT-resources MC = "" # short answer = "" # interactive/mm $ ($) ## essay " " # 3rd party SW $$ ($$) ### sketch formula

41 Exam-Scenarios design delivery tech-support IT-resources browser based one-to-one % % % # browser based open!closed %%% % % # browser based multimedia %% % %% ## online & paper % %% % # 3rd party SW %% %% %% ### CAD %% %% %% #### tablet???? 41

42 Designing Exams: What to consider! WHO do I wish to assess?! WHAT do I wish to assess?! HOW do I wish to assess? & SCENARIO

43 Selective Exams: Valid and Fair good practice core concepts vs. common fallacies differentiation transfer synthesis criticism & evaluation application bad practice unclear tasks/criteria unfamiliar tools stress, time pressure distractions & interferences pedantry & hairsplitting 43

44 Design & Implementation competencies learning objectives technological means marking scheme scenario tasks exam grading learning activities 44

45 Selective Exams: Valid and Fair good practice core concepts vs. common fallacies differentiation transfer synthesis criticism & evaluation application bad practice unclear tasks/criteria unfamiliar tools stress, time pressure distractions & interferences pedantry & hairsplitting 45

46 Cheating! prevention! authentification! ETH-Login! student card! proctoring! answers in random order! secure exam clients! Safe Exam Browser, SEB! assigned workstations! VDI! (video) log

47 Planning & Conducting an Online-Exam evaluate plan mark & grade develop conduct 47

48 Legal aspects! examination regulations:! e-assessments are a form of written assessment! right of access! filing: deletion after 2 years

49 Appeals! appeals are possible! treatment identical to other written assessments! focus of enquiry: was candidate discriminated against! no appeals so far at Online-Exams at ETHZ

50 Usability Monday, 6 October 1450

51 eas Online Conference e-assessment Scotland 2014 Online Presenter Contact: daniel.schneider[at]let.ethz.ch tobias.halbherr[at]let.ethz.ch With thanks from your hosts Professor Geoff Crisp, Dean Learning and Teaching, RMIT University Dr Mathew Hillier, Institute for Teaching and Learning Innovation, University of Queensland