Multimedia Networking Project 2b

Evaluation of Speak

Due date: Thursday, March 21st, 2013 in class


Index


Overview

You are to conduct experiments to evaluate the performance of your VoIP application (Speak) from Project 2. The focus of this project is not on how well your software works technically, but rather the scientific evaluation of the audio quality and ability to have a conversation over a range of network and system conditions. Of importance is the the design, implementation and analysis of your experiments to evaluate the performance of your VoIP application, culminating in a report (the dissemination).


Details

Design Experiments

There are two parts to evaluation of your VoIP application. The first involves an in-depth user study amongst yourself and a friend (called a "group"). The second involves a briefer user study outside your group.

For both parts, the primary means you will use to evaluate your software is by user subjective opinions of perceptual quality. You should record user opinions on a scale of (1-100 or 1-10 or 1-5 stars or something similar), based on a brief (no more than 0.5-2 minute) conversation. You are free to use whatever means you think appropriate to gather user numerical ratings (you should justify these means in your report) and you can gather additional user comments or feedback, also, if you wish.

The baseline condition you will test is: 1) a sampling rate/interval of 40ms, 2) UDP connection, 3) speech detection off, and 4) 0% loss.

In Group

You will manipulate an independent variable and determine its impact on user perception of your VoIP application. The cases you should specifically examine above your default VoIP session above (individually, not in combination) are:

You may study additional cases if you wish.

Outside Group

You will evaluate your VoIP application by running a simple user study with people outside your group.

Results and Analysis

You must provide details on both the results and the analysis. The results are the numeric measures recorded in the experiments, in the form of graphs, charts or tables. The analysis involves manipulating the data to understand relationships and interpreting the results. The analysis should consider details on the overall effectiveness of your VoIP application.

Report on Findings

The main deliverable for this project is a report describing:


Hints

Refer to project 1b hints on report and graphs. Those hints are all relevant for project 2b, too. You might look at the slides for this project (pptx).


Hand In

You must turn in a hard-copy of your project report. Please include a cover page with a title, abstract and group members. Unless otherwise specified, the hard-copy must be given to me at the beginning of class, or delivered to FL B24b before class on the day it is due.

Be sure to include names of both group members on the report, as appropriate.


Grading

The grading breakdown is as follows:

     8% Abstract

     7% Introduction

           Background (as needed)

   20% Experiment design

   60% Results and analysis

     5% Conclusions

           Other

Below is a general grading rubric:

100-90. The project clearly exceeds requirements. The experiments are carefully and thoroughly done. Measures of user perception are meaningful. Both in group and out of group experiments are well-designed, conducted and analyzed. The report is clear, well-organized and complete in all sections.

89-80. The project meets requirements. The experiments are complete. Measures of user perception are in place. Both in group and out of group experiments are complete, with results and analysis. The report is complete well-organized and complete in all sections.

79-70. The project barely meets requirements. The experiments are in place but may not be complete. Measures of user perception are in place, but are not clear or are not fully justified. Both in group and out of group experiments are in place, with results and analysis, but all may not be complete. The report is complete, but may lack some organization or may be inadequate in some sections.

69-60. The project fails to meet requirements in some places. The experiments are not complete. Measures of user perception are not clear nor well justified. Parts of the in group or out of group experiments are missing, with results and analysis correspondingly missing or incomplete. The report is lacking in organization, or may be missing some sections.

59-0. The project does not meet requirements. The experiments are not complete. Measures of user perception are missing or are not clear nor well justified. Parts of the in group and out of group experiments are missing, with results and analysis correspondingly missing. The report is lacking in organization and clarity, and is missing some sections.


Return to the Multimedia Networking Home Page

Send all questions to the staff mailing list.