See solutions by Ken Loomis. (Access restricted to the WPI domain.)

• Project Instructions: THOROUGHLY READ AND FOLLOW THE PROJECT GUIDELINES. These guidelines contain detailed information about how to structure your project, and how to prepare your written and oral reports. Note that the project guidelines specify strict page limits for your written report.

• Data Mining Technique(s): Run experiments using any (combinations) of the following techniques:

  • Pre-processing Techniques:
    • Feature selection, feature creation, dimensionality reduction, noise reduction, attribute discretization, ... .

  • Data Mining Techniques:

    • Clustering
      • Simple K-means
      • HierarchicalClusterer: Make sure to experiment with different "linkType"s

    • Anomaly Detection
      • LOF filter (see WPI level below)
      • Any data mining technique (including clustering) covered in the course that can be used to detect anomalies in the data.

  • Advanced Techniques:
    • You should consider using advanced techniques to improve the accuracy of your predictions. For instace, try ensemble methods (see Section 5.6 of your textbook), ways to deal with inbalanced classification targets (see Section 5.7 of your textbook), cost-sensitive classification, etc. But, in terms of data mining techniques, this project is restricted to the techniques listed above.
    • Any other creative ideas you have to bust model perfomance and/or to combine different models into a more powerful one.

• Dataset: We will work with the same dataset used in projects 1, 2, 3, and 4: Census-Income (also known as "Adult") Dataset available from the Univ. California Irvine (UCI) Machine Learning Repository.

  In particular,

  • Use the data in the file "adult.data", and the description of the data in the file "adult.names".
  • Use the nominal attribute salary (called >50K, .50K in the data files) as the classification target.
  • remove the fnlwgt attribute from the dataset.

• Challenges: In each of the following challenges provide a detailed description of the preprocessing techniques used, the motivation for using these techniques, and any hypothesis/intuition gained about the information represented in the dataset. Answer the question provided as well as provide the information described in the PROJECT GUIDELINES

  • Easy Level: In this part, you will use hierarchical clustering. Given that visualizing and interpreting a dendrogram over a large dataset in Weka is difficult, use a small subset of the dataset (say only 50-100 data instances) obtained using stratified random samplying. Apply hierarchical clustering to this subset. You can restrict your experiments to Euclidean distance, but make sure to experiment with different "linkType"s (single, complete, average, centroid, and Ward). Provide detailed descriptions of the parameters used to develop your model and/or preprocessing techniques used. One should be able to repeat the experiment from your description.
    1. What differences did you notice when you compared dendrograms produced using different linkTypes? Explain.
    2. Does any of these dendrograms single out potential outliers in the subset? Explain.
    3. Include visualizations that can help explain your answers to the questions above.

  • Moderate Level: Use the entire dataset for this Level. Use SimpleKMeans clustering in Weka to generate a clustering of this dataset. Be sure to IGNORE the classification attribute when performing the clustering, but DO NOT remove it from the dataset. Experiment with values for k in a systematic manner (e.g., k = 2, 4, 6, 8, 10, ... as time allows). Experiment with both Euclidean distance and Manhattan distance.
    Examine the clusterings you obtain. Answer the following questions in your description about this experiment:
    1. For each of the two distance metrics, plot a 2D graph with k (= number of clusters) on the x-axis, and SSE (= "Within cluster sum of squared errors" in the Weka output) on the y-axis.
    2. Based on these plots, which distance metric and which value for k worked best for this experiment? Justify your answer. Why do you think this was so?
    3. For this selected distance metric and value of k above, compare and contrast the assigned cluster of the instances with their classification (salary) values. Provide a visualization and a description. Were the results what you expected?

  • WPI Level: Use the entire dataset for this Level.
    Install the Local Outlier Factor (LOF) filter add-on package. Open the package manager in Weka and locate "localOutlierFactor" and install it.
    Use the LOF filter on your dataset. Identify the top 25 instances that are identified as outliers. Look at these 25 data instances and try to determine what is different about them that may make them outliers. Perform SimpleKMeans clustering on the data both with and without these instances included in the dataset. In the dataset with the instances included, change the classification value of the 25 outliers to a new class value "Outlier" (so that these instances can easily be identified in the clusters that you will construct). IGNORE the class attribute and the LOF attribute when performing the clustering. Describe the resulting clusterings.
    1. How do these two clusterings compare to each other?
    2. Attempt to use visualization to find these outliers. Were you successful?
       • a. If not: Is there some way you can characterize these outliers? Describe an idea for a method.
       • b. If so: Describe any useful information you can glean from the outliers?
    3. What challenge(s) did you encounter while performing this experiment? Give an more detailed explanation of how you used preprocessing, visualization, postprocessing or some other technique to overcome a specific challenge.
    4. Use any other anomaly detection approach you wish to identify 25 outliers in this dataset. Include these outliers in your report, and explain in detail how you found them and why they are outliers. Compare this list of 25 outliers to that you found above using the LOF filter.