A Dummy’s Guide to your Next EXPeriment: Experimental Design and Analysis Made Easy

Experiment design is a challenging task for novice and sometimes even more advanced HCI researchers. This course will introduce the field of experimental design and analysis to participants. Participants will be able to design an experiment, implement the experiment, gather data and analyze data during the course. By grounding the course in a real-life exemplar, participants will be able to have practical experience in running a controlled experiment in HCI. The course will be run by a group of HCI researchers and educators who come from a wide range of research traditions including psychology and computer science. They have worked and taught in both educational and industrial institutions.

Date of the Course: Thursday, 12 May

Time: 9:00 – 15:40

To register for the course you can follow the link below and select our course as part of your program for CHI 2016:

https://www.regonline.com/Register/Checkin.aspx?EventID=1787541

*Discounted early bird registration fee


Prerequisites

This course is open to all participants, and no prerequisites are required.


Who is this Course Suitable for?

Although the course’s title involves the term dummy’s guide, the content of the course is suitable for researchers with all levels of expertise. Beginner researchers can walk away with the basic knowledge of controlled experiment design and analysis, the ability to use two software packages for designing, running, and analysis of experiment. Intermediate and more advanced researchers can obtain a new methodology to teach controlled experiment design, and tools and shortcuts to help them to more effectively design, conduct, and analyze their experiments.


Course Content

The course has three sessions, each session lasts about 80 minutes. In the first session, we introduce a 5-step approach of experimental design. In the second session we will teach participants how to use an experiment design software (NexP) to design and conduct a known experiment. Finally, in the third session we introduce basic concepts of statistical analysis and how to perform analysis on the data they gathered.

Session 1: The 5-Step Approach In this first session, we present the five step approach and support each step with relevant examples. We also discuss potential issues that may appear for each step and how to avoid them. The five steps are:

1.Define the research question

2.Determine variables

3.Arrange conditions

4.Decide blocks and trials

5.Set instruction and procedures

Session 2: Design and running the bubble cursor experiment using NexP In session 2, we first introduce NexP, a web-based beginner-friendly experimental design and running platform that implements our 5-step approach. Participants will practice with NexP by reproducing the Pie Menu vs. Linear Menu study [1]. We then introduce a second exemplar (Bubble Cursor [2]), and ask the participants to replicate this experiment. They can even extend the existing study with more test conditions. Once the experiment is designed, we provide the Bubble Cursor code so that the participants can carry out the experiment in class with their classmates. The four instructors will facilitate the process and answer any questions raised by the participants.

Session 3: Statistical analysis We will first give a mini-lecture of basic statistical concepts, including normal distribution, central limit theorem, t-Test, and ANOVA. We then introduce a software package called JASP to the participants. Participants will be given sample data to practice various t-tests and ANOVA. After the practice, the participants will be asked to analyze the experimental data generated in session 2 and generate a report.

Learning Objective

The course has two main objectives – to help participants expand and improve their understanding of the design of controlled experiments, and to enable participants to design and implement moderately complex controlled experiments in the field of HCI.

Controlled Experiments

The first half of the course focuses on how HCI researchers create the evidentiary trial that permits a scientific claim to be made about the system in question. For instance, the course will help participants to discern between factors that affect human performance and manipulated variables that help to answer the research question. Once this is done, the participants will be able to formulate a testable, provable research hypothesis.

Design and Implementation

In our work with undergraduates, we have seen that experimental design alone is insufficient to create a true understanding of how to make an experimental claim and the proving of that claim. We believe it is important that beginner researchers implement the experiment as well. By running the test they have formulated, gathering the data, and completing the planned analysis of the data, students will gain a more thorough understanding of the decision making process that goes into designing experiments. This completes the learning loop [3]. Thus, the second objective of the course is to permit participants to run a full implementation of the experiment, with data gathered from their fellow classmates. The ensuing analysis and discussion will cement the learning.


Course Materials

Course slides are provided as additional material.

NexP: a beginner friendly software for designing and running controlled experiment http://www.nexp.site

 

JASP: an open source software for statistical analysis: https://jasp-stats.org


Course Organizers

Untitled-3Shengdong Zhao is an Assistant Professor in the Department of Computer Science, National University of Singapore. He started and currently manages the NUS HCI research lab. He has a wealth of experience in developing new interface tools and applications, and a regular contributor to the SIGCHI conference (mentioned by ACM Interaction Magazine as the most prolific author in Asia at the ACM SIGCHI conference for the last six years). He has been teaching controlled experiment design and analysis for more than 6 years at  various schools in Asia and in Europe.  http://www.shengdongzhao.com

Untitled-1Pin Sym Foong is currently a doctoral candidate at the NUS Graduate School for Integrative Sciences and Engineering. She has five years of experience as an HCI instructor at the undergraduate level, and was previously a web interface developer. http://pinsym.wordpress.com

 

Untitled-4Xiaojun Meng is currently a doctoral candidate in the Department of Computer Science, School of Computing, National University of Singapore. He has eight years of experience as a programmer. His current research work involves the end-user interface and user evaluation. He implemented the web-based toolkit NexP for designing and running controlled experiments. http://www.comp.nus.edu.sg/~xiaojun/

Untitled-2

Simon Perrault is an Assistant Professor in Yale-NUS College, a newly created liberal arts college. He is a regular contributor to the SIGCHI conference and has been teaching controlled experiment design in a common curriculum module for literature, economics and global affairs majors using the 5-step approach.


Publication

 
Shengdong Zhao, Xiaojun Meng, Pin Sym Foong, Simon Perrault. A Dummy’s Guide to your Next EXPeriment: Experimental Design and Analysis Made Easy. To appear in Proceedings of the 34th annual ACM conference on Human factors in computing systems (CHI ‘2016 Extended Abstract – Course), ACM, New York, NY, USA.