Guidelines for Projects and for Written Reports

• General Guidelines for the Experiments

  • Objectives of the Data Mining Experiments: The overall objective of the project is to obtained the most accurate prediction model you can.

  • Data and Domain Knowledge: Before you start running experiments, make sure to understand the raw data (based on the knowledge gained in Project 1) very well, to learn as much as you can about the domain, and to research approaches used by others on this dataset to the extent possible.

  • Performance metric(s): Classification accuracy is the metric that we want to maximize. But it is important to also pay attention to other metrics relavant to the data mining method under consideration. Include in your report the size of the models you construct, their readability, and how much time it took Weka to construct them.

  • Preprocessing of the Data: For each experiment, you should pre-process your data based on the knowledge you have of the data and the domain, as well as based on the results of previous experiments. For instance, you may decide to remove apparently irrelevant attributes, replace missing values if any, discretize attributes in a different way, etc. If Weka does not provide the functionality that you need to preprocess your data to obtain useful patterns, preprocess the data yourself either by writing the necessary filters (you can incorporate them in Weka if you wish), or by using a separate tool.

  • Training and Testing Instances: Use 10-fold cross-validation (unless the combined execution time is prohibiting, cross-validation is not consistent with the performance metric(s) chosen, or the particular project description asks you to use a different experimental protocol).

• Specific Guidelines for the Experiments and Written Report

  Your written report should consist of at most 9 pages distributed as follows:

  • Page 1 should contain each of the team members' names, and a description of who did what on this project.

  • [10 points] Algorithms and Code: Page 2 should provide algorithmic descriptions of the data mining techniques and of any advanced techniques that you used in the project. Read the Weka code that implements the techniques. In your written report, describe the algorithm underlying the code IN YOUR OWN WORDS. Briefly explain the algorithm in terms of the inputs it receives and the outputs it produces, AND the main steps it follows to construct the model. Make sure that the algorithms you described are the ones implemented in the Weka code (which are not necessarily the same as the ones described in the textbook or in class).

  • [60 points] Experiments: The next 1-2 pages should contain a table summarizing the experiments you ran. You should run a sufficient number of coherent experiments (that is, the results of an experiment should motivate the next experiment(s)). Each row in the table corresponds to an experiment and the columns are as following:
    1. Data: What data did you use to construct and test your model?
    2. Pre-Processing: What pre-processing was done to the data in order to improve the model's performance.
    3. Post-Processing: What post-processing was done to the model in order to improve the model's performance.
    4. Experimental Protocol: Only if for some reason it is different from 10-fold cross-validation.
    5. Resulting model: Describe the size and readability of the resulting model. Include any relavant observations about the model.
    6. Performance of the resulting model:
       • State what the performance of the model is.
       • If applicable, elaborate on the confusion matrix and/or other relevant performance indicators.
       • How long did it take Weka to construct this model?

  • [20 points] Best Performing Model. The next 2-4 pages should provide a detailed description of the experiment that yielded the best performing model you were able to construct in this project. Elaborate on the aspects listed above as columns in the experiments table. That is, include detailed descriptions of:
    1. pre-processing done,
    2. advanced techiques used,
    3. clever ideas used during the contruction of the model Provide both motivation and justification for running this experiment in the way that you did (an important aspect of this part of the report is story-telling). Provide any useful graphs and visualizations.

  • [10 points] Conclusions and Lessons Learned. The last page of your report should include a discussion of:
    • How the performance of the experiments in this project compare with that of your best performing results from previous projects on the same dataset.
    • How well the particular data mining methods used in this project worked on this dataset.
    • What combination of pre-processing, parameter values, and/or post-processing yielded particularly good results?
    • What lessons you learned in this project and how you would use these lessons on the next project.
    • Strengths and weaknesses of your project.

Guidelines for Oral Reports and Slides

Submission and Due Dates

1. Please submit the following files by email to cs4445-staff@cs.wpi.edu BY 8:00 AM the day the project is due (submissions received after 8:00 am won't be accepted):

   1. The best model you obtained in this project: Save the best model you obtained in this project onto the file

      [your-lastname]_[your-partner's_lastname]_proj[n]_best.model

      We'll run your model over our secret test dataset(s) to measure the accuracy of your model. This accuracy will be used as your entry for this project in this course kdd-cup competition.

      To save this model in Weka follow these steps:

      1. construct the model using Weka, as usual
      2. In the bottom left sub-window of Weka Explorer, under "Result list (right-click for options)", right-click the model you want to save.
      3. Click on the "Save model" option of the pop-up window.
      4. name your model file as requested in the project description:

                [your-lastname]_[your-partner's_lastname]_proj[n]_best.model
         
              even better, add to that name the challenge the model was built for:
         
                [your-lastname]_[your-partner's_lastname]_proj[n]_best_boolean.model
         
               [your-lastname]_[your-partner's_lastname]_proj[n]_best_multival.model
                

   2. Your group's oral presentation: Submit the following file with the slides for your (group's) oral report:

      [your-lastname]_[your-partner's_lastname]_proj[n]_slides.[ext]

   where: [n] is the project number; [ext] is pdf, ppt, or pptx. Please use only lower case letters in the name file. For instance, the file with my slides for Project 1 would be named ruiz_proj1_slides.ppt (if I worked on the project alone) or ruiz_smith_proj1_slides.ppt (if I worked on the project in a group with Joe Smith). Please sort the lastnames in alphabetical order.

2. In addition, each groupd should submit:
   1. a hardcopy of the individual solutions (one per student) to the homework assignment, AND
   2. a hardcopy of your group written report
   by the beginning of class the day that the project is due.

Grading Criteria

• your written report will count for 85%,
• your oral presentation will count for 10%, and
• your class participation during project presentation will count for the remaining 5% of your project grade.