Multimedia Networking Project 2b

Due date: November 17, 2004

Index

• Overview
• Details
• Hints
• Hand In

Overview

You are to conduct experiments to evaluate the performance of your audioconference (Speak) from Project 2. The focus of this project is not on how well your software works, but rather the scientific evaluation of the audioconference quality under a wide-range of system conditions. Of importance is the the design, implementation and analysis of your experiments to evaluate the performance of your audioconference, culminating in a report (the dissemination).

Details

Design Experiments

There are two parts to evaluation of your audioconference. The first involves an in-depth user study amongst your group. The second involves a briefer user study outside your group (you may have to grab a friend if you are doing the projects solo).

For both parts, the primary means you will evaluate your software is by user perception. You should record user opinions on a scale of 1-100, based on a 1-2 minute conversation. You are free to use whatever means you think appropriate to gather user numerical opinions (you should justify these means in your report) and you can gather additional user comments or feedback, also, if you wish.

The baseline condition you will test is a sampling rate/interval of 40ms, UDP connection, speech detection off, and 0% loss.

In Group

You will manipulate an independent variable and determine its impact on user perception of your audioconference. The cases you should specifically examine above your default audioconference (individually, not in combination) are:

• Loss: 0%, 1%, 5%, 10%, 20%
• Latency (by increasing the sample interval): 40ms, 100ms, 250ms, 500ms, 1000ms
• Connection type: TCP, UDP
• Speech detection: on, off

Outside Group

You will evaluate your audioconference by running a simple user study with people outside your group.

• You should find 1-3 people outside of your group to use Speak.
• Record some basic information on the users (age, gender, profession).
• Have the users run Speak under loss rates of 0%, 5%, 20% (you may provide additional cases if you wish).
• Record a perceptual quality score (scale 1-100) and some subjective comments ("sounded good", "was hard to talk", etc). Explicitly ask them to compare the quality of the Speak session with that of a typical telephone call.

Results and Analysis

You must provide details on both the results and the analysis. The results are the numeric measures recorded in the experiments, in the form of graphs, charts or tables. The analysis involves manipulating the data to understand relationships and interpreting the results. The analysis should consider details on the overall effectiveness of your audioconference.

Summarize findings

The main deliverable for this project is a report describing:

• Introduction, primarily indicating hypotheses and motivation for them.
• Background on your speech detection algorithm, providing any details relevant to understanding the implementations you test.
• Design of your experiments, providing details on all of the above, including measurement methods and independent variables chosen.
• Analysis of the results.
• Conclusions you can draw from your analysis. You should summarize your findings and decide if your data supports or rejects your hypotheses or is inconclusive.
• An abstract, summarizing your report in 1 paragraph. (You will write this last, but it should go first in the report).

Hints

Visualizations, such as graphs or charts, even simple ones, are typically much better representations of data than just tables of numbers. All graphs should include:

• A title which summarizes what the graph is showing. The title can be at the top of the graph or in the caption below the graph.

• The independent variable on the x-axis (horizontal axis) and the dependent variable on the y-axis (vertical axis). The independent variable is the one that you manipulate, and the dependent variable is the one that you observe. Note that sometimes you don't really manipulate either variable, you observe them both. In that case, if you are testing the hypothesis that changes in one variable cause changes in the other, put the variable that you think causes the changes on the x-axis.

• Labels on the x-axis and the y-axis. Where appropriate, these labels must include the units of measurement. Examples include "arm length, in cm," "time, in generations," and "number of head bumps per week."

• Numbers on the x-axis and the y-axis. These should be evenly spaced numbers, such as 0.4, 0.5, 0.6, 0.7.... If the numbers on your x-axis aren't evenly spaced (for example, they go 0.45, 0.5, 0.6, 0.62, 0.63) it probably means you chose the wrong kind of graph. When there are more than one set of points, they should be labeled (in the form of a legend/key or next to each line). If you only have one set of points, do not include a legend.

If you are using Windows, MS Excel has good support for drawing graphs. You might try this tutorial http://www.urban.uiuc.edu/Courses/varkki/msexcel/graphs/Default.html to get started.

If you are using Unix, gnuplot has good support for drawing graphs. You might see http://www.gnuplot.info/ for more information.

You might look at the slides for this project (ppt, pdf) and the slides for Experiments in Computer Science (ppt, pdf).

Hand In

You must turn in a hard-copy of your project report. Please include a title page with a title, abstract and group members. Unless otherwise specified, the hard-copy must be given to me, or delivered to FL138 on the day it is due.

Return to the Multimedia Networking Home Page

Send all questions to the Mark Claypool.