Multimedia Networking Project 2b

Due date: Thursday, November 4th, 2010 in class

Index

• Overview
• Details
• Hints
• Hand In

Overview

You are to conduct experiments to evaluate the performance of your VoIP application (Speak) from Project 2. The focus of this project is not on how well your software works technically, but rather the scientific evaluation of the audio quality and ability to have a conversation over a range of system conditions. Of importance is the the design, implementation and analysis of your experiments to evaluate the performance of your VoIP application, culminating in a report (the dissemination).

Details

Design Experiments

There are two parts to evaluation of your audioconference. The first involves an in-depth user study amongst yourself and a friend (called a "group"). The second involves a briefer user study outside your group.

For both parts, the primary means you will use to evaluate your software is by user perception. You should record user opinions on a scale of (1-100 or 1-10 or 1-5 stars or something similar), based on a brief (1-2 minute) conversation. You are free to use whatever means you think appropriate to gather user numerical opinions (you should justify these means in your report) and you can gather additional user comments or feedback, also, if you wish.

The baseline condition you will test is: 1) a sampling rate/interval of 40ms, 2) UDP connection, 3) speech detection off, and 4) 0% loss.

In Group

You will manipulate an independent variable and determine its impact on user perception of your VoIP application. The cases you should specifically examine above your default VoIP session above (individually, not in combination) are:

• Loss: 0%, 1%, 5%, 10%, 20%
• Latency (by increasing the sample interval): 40ms, 100ms, 250ms, 500ms, 1000ms
• Connection type: TCP, UDP
• Speech detection: on, off

Outside Group

You will evaluate your VoIP application by running a simple user study with people outside your group.

• You should find 1-3 people outside of your group to use Speak.
• Record some basic demographic information on the users (age, gender, profession, VoIP use).
• Have the users run Speak under loss rates of 0%, 5%, 20% and latencies of 40ms, 500ms and 1000ms (you may provide additional cases if you wish).
• Record a perceptual quality score (scale 1-10 or some similar, as above) and some subjective comments ("sounded good", "was hard to talk", etc). Explicitly ask them to compare the quality of the Speak session with that of a typical, land-line telephone call.

Results and Analysis

You must provide details on both the results and the analysis. The results are the numeric measures recorded in the experiments, in the form of graphs, charts or tables. The analysis involves manipulating the data to understand relationships and interpreting the results. The analysis should consider details on the overall effectiveness of your audioconference.

Report on Findings

The main deliverable for this project is a report describing:

• Introduction, providing a brief overview and indicating hypotheses and motivation for them.
• Background on your speech detection algorithm, providing any details relevant to understanding the implementations you test.
• Design of your experiments, providing details on how your experiments were setup, including hardware and network setup, measurement methods and independent variables chosen.
• Analysis of the results.
• Conclusions you can draw from your analysis. You should summarize your findings and decide if your data supports or rejects your hypotheses or is inconclusive.
• An abstract, summarizing your report in 1 paragraph. (You will write this last, but it should go first in the report).

Hints

Visualizations, such as graphs or charts, even simple ones, are typically much better representations of data than just tables of numbers. All graphs should include:

• A title which summarizes what the graph is showing. The title can be at the top of the graph or in the caption below the graph.

• The independent variable on the x-axis (horizontal axis) and the dependent variable on the y-axis (vertical axis). The independent variable is the one that you manipulate, and the dependent variable is the one that you observe. Note that sometimes you don't really manipulate either variable, you observe them both. In that case, if you are testing the hypothesis that changes in one variable cause changes in the other, put the variable that you think causes the changes on the x-axis.

• Labels on the x-axis and the y-axis. Where appropriate, these labels must include the units of measurement. Examples include "arm length, in cm," "time, in generations," and "number of head bumps per week."

• Numbers on the x-axis and the y-axis. These should be evenly spaced numbers, such as 0.4, 0.5, 0.6, 0.7.... If the numbers on your x-axis aren't evenly spaced (for example, they go 0.45, 0.5, 0.6, 0.62, 0.63) it probably means you chose the wrong kind of graph. When there are more than one set of points or more than one trendline, they should be labeled (in the form of a legend/key or next to each line). If you only have one set of points, do not include a legend.

If you are using Windows, Microsoft Excel has good support for drawing graphs. You might try this tutorial http://www.ncsu.edu/labwrite/res/gt/gt-menu.html to get started.

If you are using Unix, gnuplot has good support for drawing graphs. You might see http://www.gnuplot.info/ for more information.

You might look at the slides for this project (ppt, pdf) and the slides for Experiments in Computer Science (ppt, pdf).

Hand In

You must turn in a hard-copy (print out) of your project report. Please include a title page with a title, abstract and group members names. The hard-copy must be given to me (or delivered to FLB24b), before class on the day it is due.

Return to the Multimedia Networking Home Page

Send all questions to the Mark Claypool.