Due date: October 13, 2004
You are to conduct experiments to evaluate the performance of your Speech Detection algorithm from Project 1. The focus of this project is not only on how the algorithm (or sound system) performs, but on the formulation of hypotheses; the design, implementation and analysis of experiments to test the hypotheses; and a writeup of it all.
In evaluating your algorithm, there are two measures of performance you will need to consider:
User perception. Some possibilities are:
System impact. Some possibilities are:
You will decide on how each is to be measured. For example, you may choose to get a user's opinion number from 1-10 to measure user perception and you may choose to use processing time to measure system overhead. You should consider the appropriateness of your measure along with accuracy and possible sources of error.
Then, you will manipulate the independent variable and determine its impact on the algorithm through your chosen performance measures. You must pick at least two independent variables. Some possibilities are:
In addition, you must chose 1 algorithm modification and evaluate it. Possibilities include:
You should formulate hypotheses about how a change in the independent variables affects your measures of performance.
You must provide details on both the results and the analysis. The results are the numeric measures recorded in the experiments, in the form of graphs, charts or tables. The analysis involves manipulating the data to understand relationships and interpreting the results. The analysis should consider whether the data supports or rejects the hypotheses.
The main deliverable for this project is a report describing:
Good experimental writeups provide sufficient details for a knowledgeable reader to reproduce the results obtained. Keep this in mind when doing your writeup. In particular, for system measures, you should record details about the hardware and software used. For user perception measures, you should record background information on the subjects, familiarity with the topic, method sampled, etc.
Visualizations, such as graphs or charts, even simple ones, are typically much better representations of data than just tables of numbers. All graphs should include:
A title which summarizes what the graph is showing. The title can be at the top of the graph or in the caption below the figures
The independent variable on the x-axis (horizontal axis) and the dependent variable on the y-axis (vertical axis). The independent variable is the one that you manipulate, and the dependent variable is the one that you observe. Note that sometimes you don't really manipulate either variable, you observe them both. In that case, if you are testing the hypothesis that changes in one variable cause changes in the other, put the variable that you think causes the changes on the x-axis.
Labels on the x-axis and the y-axis. Where appropriate, these labels must include the units of measurement. Examples include "arm length, in cm," "time, in generations," and "number of head bumps per week."
Numbers on the x-axis and the y-axis. These should be evenly spaced numbers, such as 0.4, 0.5, 0.6, 0.7.... If the numbers on your x-axis aren't evenly spaced (for example, they go 0.45, 0.5, 0.6, 0.62, 0.63) it means you chose the wrong kind of graph. When there are more than one set of points, they should be labeled (in the form of a legend or key or next to each line). If you only have one set of points, do not include a legend.
If you are using Windows, MS Excel has good support for drawing graphs. You might try this tutorial http://www.urban.uiuc.edu/Courses/varkki/msexcel/graphs/Default.html to get started.
If you are using Unix, gnuplot has good support for drawing graphs. You might see http://www.gnuplot.info/ for more information.
You might look at the slides for this project (ppt, pdf) and the slides for Experiments in computer science (ppt, pdf).
You must turn in a hard-copy of your project report. Please include a title page with a title, abstract and group members. Unless otherwise specified, the hard-copy must be given to me, or delivered to FL138 on the day it is due.
Return
to the Multimedia Networking Home Page
Send all questions to the Mark Claypool.