PROJECT 2 - Interface Experiment
"the impact of line width"

Note: Make sure you provide all the parts requested and justify all the answers.

Due:

Project 2 is due M 3 Apr

Motivation and Goals:

The goals of this project are:

1. for you to investigate the impact of small design decisions
   on the performance of a user as they use an interface;
2. to show how the scientific method can apply to areas of Computer Science;
3. to provide you with some basic experience with "experiments";
4. to provide you with some basic data analysis experience;
5. to provide an experience of doing professional level work.

An HCI professional would be required to know what the trade-offs are between design decisions. She or he would also be required to evaluate interfaces by doing well-designed, well-controlled experiments, by doing usability studies, and by doing surveys of users.

Experiments are typically done with a large group of subjects (30 or more, for example) in order to confirm a hypothesis or to collect enough data from which a hypothesis might be infered. The large size of the subject group allows statistical tests to be carried out. In contrast, Usability Studies can be done with as few as 5 users, and are typically used to evaluate an interface design with regard to its usability. However, these two activities have much in common.

Experiments and Usability Studies do help. Many software companies these days conduct extensive usability studies on their products. As users, we should demand that usability studies be performed: software (especially non-UNIX software) is expensive, therefore it should meet our expectations. As computer scientists and software engineers we need to be aware of what can make an interface difficult for the user (e.g., due to the Gulf of Evaluation) and learn how to design and carry out usability studies.

Experiments and usability studies require hard work and good knowledge of what measurements need to be made. For example, in class we look at Fitt's law, with which we can estimate the time for a user's hand to move to a destination area, such as the movement needed to drag a file icon to the trash bin, point the cursor at a menu item using a mouse, or move your hand to the mouse in the first place. This useful result came from analysis of experimental data.

You should note that one of the challenges in performing both experiments and usability studies is to control all other factors apart from the one in which you are interested. This will help to ensure that you are producing meaningful results. This project will give you a taste of the difficulty of this problem.

Portions of the textbook, about text, layout, and evaluations, should be useful to read during this project.

Task:

In this project you:

• test hypotheses about the impact of selected text presentation changes on
  1. the reading speed,
  2. comprehension as well as
  3. user preference/belief
  for text in an interface;
• design an experiment to confirm the hypotheses; and
• do a simple statistical analysis of the results.

Reading speed and comprehension are the two "performance metrics".

The "variable" is variation of the line length .

You are to design and implement an experiment in which you carefully change the variable and measure the effects on the performance metrics. For your experiment you'll make controlled changes to the variable. {Note 1}.

You are to use text on web pages as the test situation in your experiment. Use one block of text on one web page for each variable value with which you chose to experiment. Use text on a white background. You should select a single font family. A font with serifs is recommended: i.e., use   "With"   not   "Without".

Please carefully read the General Concepts of Usability Testing web page before starting your work.

You need to change the variable enough {Note 2} for each subject (i.e., each person in your experiment) to be able to produce sensible graphs for your observations (i.e., the two user performance metrics).

Show reading speed ("speed" not "time") {Note 3} plotted against your variable.

You also need to plot a measure of comprehension {Note 4} against your variable.

For each user, after the session is over, ask them which variable setting they a) liked the best (preference), and b) read fastest (belief). You'll need to compare actual speed against which they thought was fastest, as well as record their preference. Of course you'll need to make sure that the users don't find out their actual speed during the tasks.

You'll need to do all of these steps:

1. Develop the hypotheses you are testing about user performance and preference/belief with respect to the variable
   (it helps to think of extreme situations first);
2. Describe and graph those hypotheses;
3. Identify the type of users you'll test and why;
4. Determine the design of your experiment;
5. Determine the tool(s) that you'll use for data analysis & its presentation;
6. Develop the reading tasks that your users will perform during each experiment;
7. Specify the test situation (i.e., the variations in the pages);
8. Determine how many users you'll need to test; {Note 5}
9. Select the actual users you'll test;
10. Prepare the test situation;
11. Do a trial run on 1-2 people to check your test situation & procedure, and make modifications if necessary;
12. For the real experiment, administer a short pre-test questionnaire for each user to gather relevant data;
13. Have each user sign & date a consent form agreeing to take part in the experiment, and provide an email address at which they can be contacted;
14. Describe the experimental procedure to each user in a consistent way;
15. Run each user through the tasks and collect data;
16. Do an initial analysis of your speed and comprehension data by providing descriptive statistics (i.e., mean and standard deviation) for each subject and also each setting for your variable;
17. Using all your data, produce graphs for your performance metrics;
18. Calculate the Correlation between each metric and your variable, and determine whether it is "significant" at the .05 level (alpha=.05). (i.e., the odds that the correlation is a chance occurrence is no more than 5 out of 100);
19. Do a Simple Linear Regression Analysis for your data and plot the results (also see this description) {Note 6}
20. Diagram the preference/belief results;
21. Evaluate all the results, producing conclusions: explicitly compare what you found against your hypotheses. Does the user know what's good for them?
22. Assess the possible causes of errors in the data from your experiment, and in your results.

Hand In:

The printed report you write, print and hand in should describe all the steps listed above. Include all the data/information used and gathered, describe and justify all the design decisions made about your experiment, describe the experimental procedure you followed, evaluate your results (i.e., what do they mean?), and discuss the possible flaws in your experiment.

Note that you must attach the forms that the subjects signed to your main report.

Figures in color should be printed in color.

Cite any references used.

The total report length should not be more than 15 pages (not including the attached forms). Good, clear, professional writing and presentation is required.

Notes About Grading:

• This project is harder than it looks, takes thought, and takes time.
• Just to make this clear, for some similar experimental projects in the past, grades have varied from 20% to 90%.
• We reserve the right to contact your subjects to confirm that they took part.