CS 539 Spring 2003

Computer Science Department

CS539 Machine Learning - Spring 2003
Project 8 - Rule Learning

PROF. CAROLINA RUIZ

Due Date: Monday, April 14 2003 at 10 am.

Project Description
Project Assignment
Report Submission and Due Date

PROJECT DESCRIPTION

Use sequential covering algorithms to construct sets of classification rules for each of the following problems:

Predicting the class attribute (CARAVAN Number of mobile home policies) in the The Insurance Company Benchmark (COIL 2000) dataset.
Predicting whether the income of a given person is >50K or <= 50K using the census-income dataset from the US Census Bureau which is available at the Univ. of California Irvine Repository.

PROJECT ASSIGNMENT

Read Chapter 10 of the textbook on Rule Learning in great detail.

The following are guidelines for the construction of your set of rules:
- Code: You can use EITHER one of the following two implementations of sequential-covering methods, or both.
  
  Use the Prism algorithm implemented in the Weka system.
  Read the code of the Prism classifier in Weka in great detail. You can find a description of this algorithm in Section 4.4 (and variations of this algorithm in Section 6.2.) of "Data Mining: Practical Machine Learning Tools and Techniques with Java Implementations". I. Witten and E. Frank. Morgan Kaufmann Publishers. (Available on reserve at the WPI Gordon Library.)
  
  Use the FOIL system developed by Quinlan. Several versions of FOIL are available online, for instance the one available at Quinlan's Webpage or a more recent one if you find it. Quinlan's webpage contains several copies of his papers describing the techniques employed in the system.
- Training and Testing Instances:
  1. For the The Insurance Company Benchmark, use the ticdata2000.txt data for training and the ticeval2000.txt data for testing. You may restrict your experiments to a subset of the instances IF Weka cannot handle your whole dataset. But remember that the more accurate your neural network is, the better.
  2. For the Census Income data, use the census-income dataset for training. You can restrict your experiments to a subset of the dataset if your system cannot handle the whole dataset. But remember that the more accurate your system is, the better. Also, note that this dataset has missing values. It is up to you how to fill in appropriate data for those missing values. Also, it is up to you to decide if it's a good idea to discretize continues attributes, and if so, how.
    Note that Test data are also available at the UCI.
- Preprocessing of the Data: You should apply relevant filters to your dataset as needed before doing the mining and/or using the results of previous mining tasks. For instance, you may decide to remove apparently irrelevant attributes, replace missing values if any, discretize attributes in a different way, etc. Your report should contained a detailed description of the preprocessing of your dataset and justifications of the steps you followed. If Weka does not provide the functionality you need to preprocess your data as you need to obtain useful patterns, preprocess the data yourself either by writing the necessary filters (you can incorporate them in Weka if you wish).
- Evaluation and Testing: As usual, experiment with different testing methods:
  1. Supply separate training and testing data.
  2. Experiment with several split ratios.
  3. Use n-fold crossvalidation to test your results. Experiment with different values for the number of folds.

REPORT AND DUE DATE

Written Report.
Your report should contain the following sections with the corresponding discussions:
1. Code Description: Describe the Prism code that you used from Weka and/or the algorithm implemented in FOIL. Explain the algorithm underlying the code in terms of the input it receives and the output it produces, and the main steps it follows to produce this output.
2. Data: Describe the dataset that you selected in terms of the attributes present in the data, the number of instances, missing values, and other relevant characteristics.
  Provide a detail description of the preprocessing of your data. Justify the preprocessing you apply and why the resulting data is the appropriate one for mining neural networks from it.
3. Experiments: For each experiment you ran describe:
  - Data: What data did you use to construct and test your classifier?
  - Any additional pre or post processing done to the data or the classifier's output in order to improve the accuracy of your classifier.
  - Accuracy of the resulting classifier.
  - Discuss how this accuracy compares with that of your most accurate ZeroR experiment, decision trees, neural nets, Bayesian methods, instance-based learning, regression methods, and genetic algorithms from the previous assignments.
4. Summary of Results
  - What was the accuracy of the most accurate hypothesis (set of rules) you obtained?
  - Discuss how this accuracy compares with that of your most accurate results from the previous assignments.
  - Include the most accurate hypothesis you obtained in your report.
  - Discuss the strengths and the weaknesses of your project.
Oral Report. We will discuss the results from the individual projects during the class on April 14. Your oral report should summarize the different sections of your written report as described above. Each of you will have 5 minutes to explain your results and to discuss your project in class. Be prepared!
Submission and Due Date.
Please submit the following files by email to ruiz@cs.wpi.edu by the deadline specified below. Submissions received on Monday April 14, between 10:01 am and 10:45 am will be penalized with 30% off the grade and submissions after 10:45 am won't be accepted.
1. [your-lastname]_proj8_slides.[ext] containing your slides for your oral report. This file should be either a PDF file (ext=pdf) or a PowerPoint file (ext=ppt). Please use only lower case letters in the name file. For instance my file would be named ruiz_proj8_slides.ppt
2. [your-lastname]_proj8_report.pdf containing your written report in PDF.
  For instance my file would be named ruiz_proj8_report.pdf ***** ALSO, PLEASE BRING A HARDCOPY OF YOUR REPORT TO CLASS ON APRIL 14, 2003. ****

CS539 Machine Learning - Spring 2003 Project 8 - Rule Learning

PROF. CAROLINA RUIZ

PROJECT DESCRIPTION

PROJECT ASSIGNMENT

REPORT AND DUE DATE

CS539 Machine Learning - Spring 2003
Project 8 - Rule Learning