Principles of Interaction

Principles of Interaction Prof. Dr. Matthias Kranz, Priv.-Doz. Dipl.-Ing. Dr. Andreas Riener with contributions from Stefan Diewald, Patrick Lindemann, Andreas Möller, Marion Koelle, Luis Roalter, Tobias Stockinger with material partially from Andreas Butz, Paul Holleis, Albrecht Schmidt

Deployment Phase User Studies Qualitative testing Quantitative testing Laboratory testing vs. field testing Hypotheses, dependent and independent variables Study design Questionnaires and Scales Statistics Basics terms Tests Reporting and presenting results Source: http://www.mprove.de/script/00/upa/_media/upaposter_11x17.pdf 4

Deployment Phase User Studies Qualitative testing Quantitative testing Laboratory testing vs. field testing Hypotheses Study design Questionnaires and Scales Statistics Basics terms Tests Reporting and presenting results Source: http://www.mprove.de/script/00/upa/_media/upaposter_11x17.pdf 5

USER STUDIES 6

Motivation If you want to evaluate a tool, say an axe, you might study the design of the bit, the weight distribution, the steel alloy used, the grade of hickory in the handle, etc., or you may just study the kind and speed of the cuts it makes in the hands of a good axeman. Michael Scriven, 1967 Source: Jumsoft, Keynote Stills 7

Evaluation of User Interfaces Analytic Evaluation Empirical Evaluation þ Literature Review þ Model-based Evaluation þ Cognitive Walkthrough þ Heuristic Evaluation Qualitative þ Think Aloud, q Observation, q Retrospective Testing q Diary Studies Quantitative q Controlled experiments, q Physiological measurements q Questionnaires 8

ANALYTIC Q: How to evaluate a design/interface without users? 9

Model-based Evaluation Using cognitive architectures in user interface design is a form of usability evaluation based on models of the user Such models need to include critical features of human psychological limitations and abilities in the form of constraints on what the model can do Use model predictions instead of user testing 1. Describe the interface design in detail. 2. Build a model of the user doing the task. 3. Use the model to predict execution or learning time. 4. Revise or choose design depending on prediction. Models of the user can partially eliminate the need for empirical user testing of interface designs, with potential savings in cost and time. Some user testing still required (assess aspects not dealt with by an analytic model) Three general approaches to models are task networks cognitive architectures, and GOMS models 10

Lehrstuhl für Informatik mit Schwerpunkt Eingebettete Systeme, Universität Passau A. Riener, Perceptual Computer Science: Human-centric and reality-based human-machine interaction, JKU Linz, Altenberger Strasse 69, 279 pages, Habilitation thesis, approved Feb. 14, 2014. Model-based Evaluation Increasing Interaction Complexity Basic model of detecting incoming stimuli via sensory receptors no longer suitable 11

Model-based Evaluation A. Riener, Perceptual Computer Science: Human-centric and reality-based human-machine interaction, JKU Linz, Altenberger Strasse 69, 279 pages, Habilitation thesis, approved Feb. 14, 2014. Increasing Interaction Complexity Include the role of individuality, emotion, and cognition in affecting HMI 12

Heuristic Evaluation Is a usability inspection method for computer software that helps to identify usability problems in the user interface (UI) design. It involves evaluators examining the interface and judging its compliance with recognized usability principles. Heuristic evaluation debugs design. Examples Nielsen s heuristics Gerhardt-Powals cognitive engineering principles Weinschenk and Barker classification 13

Cognitive Walkthrough Formalized way of imagining people's thoughts and actions when they use an interface for the first time. Usually performed by expert in cognitive psychology. Stages 1. Select a task that the design is intended to support; for each task walkthrough considers what impact will interaction have on user? what cognitive processes are required? what learning problems may occur? 2. Try to tell a believable story about each action a user has to take to do the task. 3. To make the story believable, you have to motivate each of the user's actions, relying on the user's general knowledge and on the prompts and feedback provided by the interface. if you can't tell a believable story about an action, then you've located a problem with the interface 14

Cognitive Walkthrough What is it good for? Reflect about what the users will be thinking. Identify controls that may be missing or hard to find. Note inadequate feedback. Suggest difficulties with labels and prompts. Purpose Focus most clearly on problems that users will have when they first use an interface, without training. Not a technique for evaluating the system over time (e.g., how quickly a user moves from beginner to intermediate). Most effective if designers can really create a mental picture of the actual environment of use. The analysis focuses on users goals and knowledge: does the design lead the user to generate the correct goals? 15

Cognitive Walkthrough How to do it? Prior to doing a walkthrough, you need four things: 1. You need a description of a prototype of the interface. It doesn't have to be complete, but it should be fairly detailed. Things like exactly what words are in a menu can make a big difference à remember, e.g, Balsamiq 2. You need a task description (for a representative task). 3. You need a complete, written list of the actions needed to complete the task. 4. You need an idea of who the users will be and what kind of experience they'll bring to the job. Caveats The walkthrough does not test real users on the system. With a walkthrough you can potentially evaluate the interface by imagining the behavior of entire classes of users, not use one unique user. 16

Guidelines Inspection (for Consistency) Written guidelines recommended for larger projects Screen layout Appearance of objects Terminology used Wording of prompts and error messages Menu s Direct manipulation actions and feedback On-line help and other documentation Etc. 17

QUALITATIVE 18

Qualitative Tests Surveys Variants pre-survey pre-postcondition survey post-survey Need careful design what information is required? how are answers to be analyzed? should be PILOT TESTED for usability! Styles of questions general open-ended scalar multiple choice ranked Tools available: Google Forms, SurveyMonkey, etc. 19

Qualitative Tests (continued) Interviews (semi-structured, structured) Analyst questions user on one to one basis, usually based on prepared questions informal, subjective and relatively cheap Advantages can be varied to suit context issues can be explored more fully can elicit user views and identify unanticipated problems Disadvantages very subjective time consuming 20

Qualitative Tests (continued) Experience sampling methodology (ESM) A technique from psychology Questionnaires, filled out at various points throughout the day asking about the current experience Small set of question Different forms: Event triggered, random,.. Diaries Discipline of the user to do so, when is reported directly or at night? Observation E.g. using cameras, microphones, taking notes, 21

User Studies Qualitative Testing Observation Observation as complementing activity to interviews and questionnaires Problems solved by observation: Real-world activities are too complex to be described in relevant details Some activities are hard to describe in words Activities require cooperation with other people or events Sometimes, activities in the real-world are carried out differently from what the official procedures demand, the interviewee might be reluctant to describe the reality Observation is targeted at resolving these problems! Simplest form of observation: Ask: Can you show me how you do that? Designing Interactive Systems, D. Benyon, P. Turner and S. Turner, 2005, Addison-Wesley, chapter 9 http://www.designdownloader.com/i/?id=telescope_f003-20111205145903-00003 22

User Studies Qualitative Testing Observation Difference between observation and ethnography Ethnography observe the user in his habitat without disturbing it. e.g., work of a detective, candid camera immerse into the context. more focused on experiences than on description. Observation a method (amongst others) used in ethnography. aims to generate descriptions of the is-state. not drawing conclusions or reflecting on the real situation. observation aims to eliminate the distance between observer and observed. Designing Interactive Systems, D. Benyon, P. Turner and S. Turner, 2005, Addison-Wesley, chapter 9 http://www.designdownloader.com/i/?id=telescope_f003-20111205145903-00003 23

User Studies Qualitative Testing Observation Methods Observation methods Test monitoring (direct observation) observe and record test participants behavior test as specifically set up situation usually done with only a few participants observe and record test participants behavior use a data sheet or database to help guide the recording forms include test time, checklists of actions to be completed by participants, comments from participants (or from the test monitor), narrative descriptions of the event. Direct recording (indirect observation) eliminates monitor s potential bias audio or video recording, screen recording problem: software does not record the participant s comments, his/her workload or frustration Designing Interactive Systems, D. Benyon, P. Turner and S. Turner, 2005, Addison-Wesley, chapter 9 https://www.utexas.edu/academic/ctl/assessment/iar/tech/gather/method/use-types.php http://www.designdownloader.com/i/?id=monitor_fa008-20111110220415-00001 http://www.designdownloader.com/i/?id=news_f006-20111107173221-00003 24

User Studies Qualitative Testing Observation Methods Observation methods Think aloud (direct observation) user observed while performing a task participants are asked to comment aloud on their actions describe what he/she is doing and why verbalize thoughts what he/she thinks is happening; what could be problems allows users to freely express themselves Advantages simplicity - requires little expertise can provide useful insights can show how the system is actually used Disadvantages subjective difficult to conduct act of describing may alter task performance Designing Interactive Systems, D. Benyon, P. Turner and S. Turner, 2005, Addison-Wesley, chapter 9 https://www.utexas.edu/academic/ctl/assessment/iar/tech/gather/method/use-types.php http://www.designdownloader.com/i/?id=volume-20110816150059-00018 25

User Studies Qualitative Testing Observation Goals of Observation Seeing as many as possible situations Seeing unusual situations, when and where things go wrong Important Think about what has not been observed to avoid overgeneralization Observer can interact and focus attention towards specific activities or aspects Time to stop observation As soon as no new information appears Designing Interactive Systems, D. Benyon, P. Turner and S. Turner, 2005, Addison-Wesley, chapter 9 http://www.designdownloader.com/i/?id=icon-aluminum-20110812093346-00039 26

User Studies Qualitative Testing Observation Planning observations 1. Establish objectives and information requirements Should the coverage be in breadth or in depth? Decide on goals and usage of captured material and results first to tailor the whole process to the requirements of those who will receive the results. 2. Gain co-operation of contacts, get permissions to record Establish the times, places, and people who will be observed. Get consent of all involved persons and entities 3. Decide on the recording technique Hand-written notes, audio, video recordings, Keep in mind the reviewing time of the recorded material! It is useful to be able to make some kind of first-cut analysis during observation. Designing Interactive Systems, D. Benyon, P. Turner and S. Turner, 2005, Addison-Wesley, chapter 9 http://www.usabilitynet.org/tools/userobservation.htm 27

User Studies Qualitative Testing Observation Running observations Make sure that those being observed are aware of the reason for your study and that they do not see you in negative terms. Run a pilot observation session to get a feel for what to expect and to test out any observation sheets. This will also help to judge how long the observation session needs to be. Make sure that there is enough time for this. Try to be as unobtrusive as possible. Do not let yourself or your equipment get in the way. Note down any events that you do not understand and try to clarify them with the user as soon as the session is completed. Try to be aware of the range of influences that are affecting the user. If possible photograph the users work area or the area of operation as this will act as a reminder of the environmental context. After your observations, write down your first impressions before the analysis stage later on. Designing Interactive Systems, D. Benyon, P. Turner and S. Turner, 2005, Addison-Wesley, chapter 9 http://www.usabilitynet.org/tools/userobservation.htm 28

User Studies Qualitative Testing Retrospective Testing Retrospective Testing Recorded usability testing session, e.g., video taped prerequisite: all relevant interactions can be and have been recorded during the test Review of material by observer and the participant questioning the participant regarding their behaviour in the test Useful for situations where interaction between observer and participant is not possible during the test situation Time consuming Joint review of the recording doubles the testing time (at least) Different methods Retrospective think aloud (RTA) Retrospective think aloud (RTA) with screen video Retrospective think aloud (RTA) with gaze overlay on the recording http://www.tobii.com/global/analysis/training/whitepapers Tobii_RTA_and_EyeTracking_WhitePaper.pdf https://lh3.googleusercontent.com/ lowtf2z4nox8xure0gtnzrw_4zydkhobw0_x2vsmkgika0js1mgb7yz48j7h5nmfrgsfqutjfiocquvqqkal caqr764lv4emgdifojc43_twxun4zfq 29

User Studies Qualitative Testing Diary Study A study that asks people to keep a diary, or journal, of their interactions with a computer system/interface/device any significant events or problems during their use of a system, or other aspects of their working life, should be noted down A diary typically asks a user to record the date and time of an event, where they are, information about the event of significance, and ratings about how they feel, etc. http://www.designdownloader.com/i/?id=notebook01-20110830215502-00015 Definition from glossary http://www.usabilityfirst.com/, last visited 20.03.2012 30

User Studies Qualitative Testing Diary Study Example The beginning of a diary log sheet for one day. The participant records activities on the left as the proceeds. The research assigns categories during the end-of-day debriefing. Image from: John Rieman. The diary study: a workplace-oriented research tool to guide laboratory efforts. Proceedings of the SIGCHI conference on Human factors in computing systems. pp 321-326. 1993. 31

User Studies Qualitative Testing Diary Study Example What are potential problems with this study type? How can technologies such as voice recorders, cameras, mobile phones help? The beginning of a diary log sheet for one day. The participant records activities on the left as the proceeds. The research assigns categories during the end-of-day debriefing. Image from: John Rieman. The diary study: a workplace-oriented research tool to guide laboratory efforts. Proceedings of the SIGCHI conference on Human factors in computing systems. pp 321-326. 1993. 32

Why do we use diaries in HCI research? What is a diary study good for? Fills the gap between observations in naturalistic settings, lab studies, and surveys. Can record user-defined data E.g. eureka moments = moments when the user realizes that he/she had learned something new about the computer interface. (Rieman 1993) Attention It is extremely hard for users to keep a diary over a longer period of time!! Try it yourself: e.g. to log all your food consumption for 6 weeks or longer (4 is still ok ). Try it, e.g. www.fddb.de (nutrition diary) An interesting alternative for making diary entries is to give users a tape recorder and a list of questions, so that users don't need to write things down as they encounter them. 33

Diary study - Discussion Strengths Suitable for non-workplace, noncontrolled or on-the-go settings Understanding the why of user interaction with a technology Collecting data that is fluid, and changes over time (e.g. mood or perception) Time recording is more accurate than e.g. in a survey. Collecting user-defined data (e.g. intentions) Weaknesses Requires the participants to be aware of their actions: otherwise they may have trouble recording it in a diary entry. Participants may not follow through and record a sufficient number of entries. Time recording might be less accurate than in a controlled laboratory setting. More intrusive than e.g. a survey: it is harder to recruit participants. Table adapted from: Lazar, Jonathan, Jinjuan Heidi Feng, and Harry Hochheiser. Research Methods in Human-Computer Interaction. John Wiley & Sons, Chichester, 2010. http://www.evolllution.com/wp-content/uploads/2012/08/strengths-weaknesses-sized.jpg 34

Observations and Protocols What are observations and protocols good for? Demonstrating that a product improves productivity Basis for qualitative and quantitative findings Hint Minimize the chance for human error in observation and protocols Most people are pretty bad at doing manual protocols! Combine with computer logging Log what you get from the system Observer makes a protocol on external events Automatic and unobtrusive Large amounts of data difficult to analyze Logging on a user s notebook might be coarse and subjective vs. useful insights (usage behavior) good for longitudinal studies (couple of weeks or even longer) 35

Videoprotocols Accurate and realistic Needs special equipment for recording (Normally) obtrusive (except for candid cameras but: not allowed ) Transcription and categoriatzion of (audio and) video difficult and requires skill Some automatic support tools available (e.g., Videoobserver) Goal Valid results, e.g., for accuracy and efficiency pre-identified task outcomes (task success, task failure) time stamps in the video à calculate efficiency data satisfaction results (participant answers satisfaction questions) 36

Retrospective Testing What it is: Shown on the example of an Antivirus program Use an old (maybe archived) version of a computer program/interface here: anti-virus program and test it against situations that were not known at the date of the last product placement Here: most current viruses to find out how good (robust, fit for the future ) a program is Here: how good the heuristic and generic detection of an antivirus program really works Expected to be better than using self-written viruses (tested at current ver.) The main critical points Retrospective testings shows only something about the past, but nothing for the future that s wrong! (analogy: why have we learned history at school?) we should learn from the past (good and bad points) for improvements in the future 37

QUANTITATIVE 38

User Studies Two important types of studies Laboratory studies tight boundaries for the experiment, control, and task completion easier observation and logging potentially less suited for mobile systems and longitudinal observations Field studies increased realism in-situ study (richer environment) complex more challenging settings loss of control Lab Studies User Studies Field Studies 40

Quantitative Data Collection In lab studies, it is easy to perform measurements In field studies: Data logging You must have a plan about how you will use the log data Risk of forgetting to log something important! Logging too much can create an analysis nightmare! Make a list of questions you expect to answer with log data How many times did participants upload a photo? How many days did participants use your prototype? Important Verify that you can detect/recognize/discriminate the interesting aspects using your data using a pilot! 41

Usability Evaluation Seldom applied in reality Developers not familiar with techniques Fear of additional cost Running out of time (release deadlines) Necessity is neglected ( the product works ) What I understand, understands everyone mindset http://www.dilbert.com/strips/comic/2002-05-11/ 42

User Studies Quantitative Testing Controlled Experiment What is a usability experiment? Usability testing in a controlled environment, including Test set of users ideally selected from a database of volunteers, according to specified criteria Pre-specified tasks Data collection (quantitative and qualitative) Data evaluation (e.g. statistics) Comparison to specified goals or competing designs/systems Contrasted with evaluation methodologies such as expert review field study 43

User Studies Quantitative Testing Controlled Experiment Experimental (success) factors Subjects representative/non-representative sufficient sample for statistics? Variables independent variable (IV) characteristic changed to produce different conditions. e.g. interface style, number of menu items. dependent variable (DV) characteristics measured in the experiment e.g. time to perform task, number of errors. http://ttlearning.com/wp-content/uploads/2013/01/sat-subject-test-literature.png 44

User Studies Quantitative Testing Controlled Experiment Experimental (success) factors Hypothesis Prediction of outcome framed in terms of IV and DV Null hypothesis states no difference between conditions the aim is to disprove this https://hamlinclassof2019.wikispaces.com/file/view/ab-testing-problem-hypothesis-fullsize.png/399348028/320x450/ab-testing-problem-hypothesis-full-size.png 45

User Studies Quantitative Testing Controlled Experiment Experimental (success) factors Experimental design between groups design (each subject performs under only one condition) + no transfer of learning - more subjects required (therefore more costly) - user variation can bias results within groups design (each subject performs experiment under each condition) - transfer of learning possible + fewer subjects needed + less likely to suffer from user variation 46

Between Subjects Testing vs. Within Subjects Testing Example: Two interfaces, interface A and interface B Between subjects testing Half of the participants use interface A, half use interface B Participants each in different condition avoidance of learning effect differences between users can play a significant role, more users needed Within subjects testing All of the participants use interface A, half use interface B Each participants experiences all conditions effect of differences between users is lessened fewer participants required Attention on repetition, carryover effects and learning effects counterbalancing, Latin square, etc. 47

LAB VS. FIELD 48

Lab versus Field studies ü controlled environment ü special equipment available for conducting lab experiments ü repeatable (e.g. traffic/pedestrians in a driving simulator) lack of context not a real situation, e.g., no danger for the driver on crashing a car Results often not transferable to real life situation difficult to observe several users cooperating Appropriate if field study is too dangerous for subjects, impractical or expensive ü natural environment ü context retained (though observation may alter it) ü longitudinal studies possible, e.g. change of driving behavior over 5 years ü realistic setting distractions noise costly (not only monetary) many influencing factors (compensate with larger sample?) Appropriate for beta testing where context is crucial 49

Lehrstuhl für Informatik mit Schwerpunkt Eingebettete Systeme, Universität Passau Field Studies http://www.field-studies-council.org/media/799243/hompepage-geography.jpg Field studies are often the most appropriate method for studying people s use of UbiComp technologies, [ ] they allow to collect data [and] observe the unexpected challenges participants may experience. Consolvo et al, 2007; Rogers et al., 2007 50

Field Studies Conducted outside the lab and outside controlled environment Can be necessary in order to understand user needs or technology usage in a specific domain (e.g. emergency response, homes, context-aware systems) http://jerrodlarson.squarespace.com/picture/field%20obs.png?pictureid=17988239&asgalleryimage=true 51

Observations/Field Studies What are the best methods for analyzing the data obtained from field studies? The answer can vary greatly what kind of questions should be answered? which types of data were obtained in the field studies? Typically, the questions to be answered should dictate both the data (and with it, experimental design) and the analysis techniques that are employed Focusing on and segmenting users based on their goals, behaviors, and attitudes is often far more effective at identifying key differences (e.g., demographics: age, gender, nationality, etc.) that can be used effectively Additional readings E. Goodman, M. Kuniavsky, A. Moed, Observing the User Experience: A Practitioner s Guide to User Research, pp. 608, September 2012, Morgan- Kaufmann, ISBN: 978-0-12-384869-7 Jon Kolko, Exposing the Magic of Design: A Practitioner's Guide to the Methods and Theory of Synthesis, pp. 208, January 2011, Oxford Univ. Press, ISBN: 978-0-19-974433-6 http://www.uxmatters.com/mt/archives/2010/09/analyzing-data-from-field-studies-wiki-design.php 52

Studies of Current Behavior Goal Explore how people use existing technology in their lives, now and here Contribution Understanding of current behavior and implications for future technology Research questions for this study type emphasize how people actually use (and not report to use) technology and my be very open ended. Data collection methods Interviews Logging Observations, etc. One way of doing it Ethnography http://www.agritv.org/green/images/technology_icon.jpg 53

Proof of Concept Field Studies Goals To validate feasibility of an approach or prototype in a real-world environment or setting Etc. Contribution Technological advance rather than field study itself Technology oriented Validation of novel algorithms, approaches, concepts http://www.presentations.co.uk/wp-content/uploads/2012/10/proof-of-concept-300x178.jpg 54

Phases of a Field Study Goal Definition Study Design Participants and Task Selection Study Beta Test & Conduction Data Collection and Analysis 55

Goal Definition Determine the research question What questions do you want to answer? What will be your contribution to the research community? What are the lessons learnt and the implications? Examples of a research question Is the mobile phone a suitable proxy for the owner s location? Does context-aware power management have the potential to save energy? The guiding research question will influence many (all?) parameters, e.g., Type of participants to recruit Time frame for the study Type of data to collect 56

Study (Experiment) Design After having set the research question: think about how to answer it Important Questions 1. What type of participants you should recruit? 2. What will your participants do during the study? - Give them hardware? Give them tasks? 3. What data will you collect? 4. How long will the study be? 5. What has absolutely to work / Where can you skimp? Field study characteristics In-situ ( not on your turf") Trading control for realism - Think carefully if this is important! Often it is a good idea to do a lab study before a field study Actual practice in many of our MS thesis! 58

Study (Experiment) Design The experiment should be set up to be reproducible! Main factors Participants Independent variables Hypotheses stated Approach State the hypotheses What do you want to proof? Find the variables Which are varied? which are measured? Find participants Representative for the experiment! Fix the method to use (between-groups / within groups) 59

Testing in the Wild Mobile Apps Google Play Beta-Testing https://support.google.com/googleplay/android-developer/answer/3131213?hl=en 64

HYPOTHESES 65

Hypotheses Statement, Prediction of the result of an experiment Stating how a change in the independent variables (IV) will effect the measured dependent variables (DV) Null hypothesis States no difference between conditions The aim is to disprove this With the experiment it can be shown that the hypothesis is correct Usual approach Stating a null-hypotheses (this predicts that there is no effect of the change in the independent variable on the measured variable) Carrying out the experiment and using statistical measures to disprove the null-hypotheses When a statistical test shows a significant difference it is probable that the effect is not random 66