Due date: September 30, 2010 in class
You are to conduct experiments to evaluate the performance of your Speech Detection algorithm implementation from Project 1. The focus of this project is not only on how the algorithm (or sound system) performs, but on the formulation of hypotheses; the design, implementation and analysis of experiments to test the hypotheses; and a writeup of it all.
You are allowed to work in groups of 2 for this project. If you do work in a group, you may evaluate the implementation from either group member (or both!).
In evaluating your algorithm, there are two measures of performance you need to consider:
User perception. Some possibilities are:
System impact. Some possibilities are:
You will decide on how each is to be measured. For example, you may choose to get a user's opinion number from 1-10 to measure user perception and you may choose to use processing time to measure system overhead. You should consider the appropriateness of your measure along with accuracy and possible sources of error.
Then, you will manipulate the independent variable and determine its impact on the algorithm through your chosen performance measures. You must pick at least two independent variables. Some possibilities are:
In addition, you must chose 1 algorithm modification and evaluate it. Possibilities include:
You should formulate hypotheses about how a change in the independent variables affects your measures of performance.
You must provide details on both the results and the analysis. The results are the numeric measures recorded in the experiments, in the form of charts or tables. The analysis involves manipulating the data to understand relationships and interpreting the results, often in the form of graphs. The analysis should consider whether the data supports or rejects the hypotheses.
The main deliverable for this project is a report describing:
Note, the above bullets break naturally into Sections in the report.
Good experimental writeups provide sufficient details for a knowledgeable reader to reproduce the results obtained. Keep this in mind when doing your writeup. In particular, for system measures, you should record details about the hardware and software used. For user perception measures, you should record background information on the subjects, familiarity with the topic and type of software, method sampled, etc. For measurements, you should record details on the tools used, the process of gathering data, etc.
Visualizations, such as graphs or charts, even simple ones, are typically much better representations of data than just tables of numbers. All graphs should include:
A title which summarizes what the graph is showing. The title can be at the top of the graph or in the caption below the figures (do not use both).
The independent variable on the x-axis (horizontal axis) and the dependent variable on the y-axis (vertical axis). The independent variable is the one that you manipulate, and the dependent variable is the one that you observe. Note that sometimes you do not really manipulate either variable, you observe them both. In that case, if you are testing the hypothesis that changes in one variable cause changes in the other, put the variable that you think causes the changes on the x-axis.
Labels on the x-axis and the y-axis. Where appropriate, these labels must include the units of measurement. Examples include "arm length (cm)" "time (seconds)" and "number of head bumps per week."
Numbers on the x-axis and the y-axis. These should be evenly spaced numbers, such as 0.4, 0.5, 0.6, 0.7.... If the numbers on your x-axis are not evenly spaced (for example, they go 0.45, 0.5, 0.6, 0.62, 0.63) it means you chose the wrong kind of graph. When there are more than one set of points, they should be labeled (in the form of a legend or key or next to each line). If you only have one set of points, do not include a legend.
If you are using Windows, Microsoft Excel has good support for drawing graphs. You might try this tutorial http://www.ncsu.edu/labwrite/res/gt/gt-menu.html to get started.
If you are using Unix, gnuplot has good support for drawing graphs. You might see http://www.gnuplot.info/ for more information.
You might look at the slides for this project (ppt, pdf) and the slides for Experiments in Computer Science (ppt, pdf).
You must turn in a hard-copy of your project report. Please include a cover page with a title, abstract and group members. Unless otherwise specified, the hard-copy must be given to me at the beginning of class, or delivered to FL B24b before class on the day it is due.
Be sure to include names of all group members on the report.
You will use email to turn in any testing Code/Scripts used/modified, and any Makefile/Project file:
Package up (with pkzip) your files, for example:
mkdir proj1b copy * proj1b // copy all your files to submit to proj1b directory pkzip proj1b.zip proj1b
Attach proj1b.zip to an email to claypool@cs.wpi.edu with "cs529-proj1b" as the subject.
Return to the Multimedia
Networking Home Page
Send all questions to the Mark Claypool.