• contact-lenses.arff
• Subset of contact-lenses.arff
• contact-lenses.csv

• credit_history.arff used in the Decision Trees handout
• credit_history.arff used in the Bayesian Models handouts

• diabetes.arff
• diabetes.dat
• diabetes_no_attribute_names.dat

• iris.arff

• cpu.arff
• cpu.with.vendor.arff

• weather.nominal.arff
• weather.numeric.arff

• world_happiness_dataset.txt
• world_happiness_with_continents.arff
• small_world_happiness.arff

• Protein Datasets: These protein datasets are maintained by Shuiwang Ji, CS Department, Louisiana State University/USA.

Data Mining in Bioinformatics Series: Advanced Information and Knowledge Processing by Wang, J.T.L.; Zaki, M.J.; Toivonen, H.; Shasha, D.E. (Eds.) 1st Edition., 2005, XI, Springer.

• Weka's source code: available in a file called "weka-src.jar", located in the directory where Weka was installed. You need to unzip and/or use jar utilities on this file to extract its contents. Inside, you will find the .java files that implement Weka.
• The WEKA workbench" (online appendix)
• Consult the "README" file, the "documentation" webpage, and the "WekaManual.pdf" provided with the Weka system (in the same directory where Weka was downloaded). Browse through the "Package Documentation" to become familiar with it.
• When needed, use the following command to increase the amount of main memory used by Weka. Here, I'm increasing the amount of main memory used by Weka to 768m, but you can specify any other size instead of 768 if more memory is needed/available:

      java -Xmx768m -jar weka.jar
  

• There are several YouTube videos with information about Weka - some of them from one of the Weka creators (Ian Witten).

• Witten's and Frank's textbook slides - Chapter 2.

The main objective of this lecture is to become familiar with the Weka system.

1. WekaTextbook slides provided above.

2. Work with the Weka System

   1. Get familiar with the sample datasets provided with the Weka system.

   2. Open the contact lenses dataset contact-lenses.arff with a word editor. The file consists of:

      • An OPTIONAL header with general information about the dataset. Each line in this header starts with "%" to denote that these are commented lines.

      • The name of the dataset

          @relation contact-lenses
                                   

      • The name and types of the attributes in the dataset

          @attribute age                  {young, pre-presbyopic, presbyopic}
          @attribute spectacle-prescrip   {myope, hypermetrope}
          @attribute astigmatism          {no, yes}
          @attribute tear-prod-rate       {reduced, normal}
          @attribute contact-lenses       {soft, hard, none}
                                   

        These attributes are the "column names" of the tabular data that comes after.

      • The dataset instances

          @data
        
          young,myope,no,reduced,none
          young,myope,no,normal,soft
          young,myope,yes,reduced,none
          young,myope,yes,normal,hard
           ...
        
                           

3. Run the Weka system. Choose the "Explorer" option in the main window.

4. Load the contact lenses dataset into the Weka System using "Open File".

5. Apply the following classification techniques (under "Classify") to this dataset

   • ZeroR: 0 predicting attributes are used to construct a classification rule

   • OneR: 1 predicting attribute is used to construct classification rules

   • J48: Decision Trees

6. Go back to the "Preprocess" tab and modify the input dataset using the following preprocessing filters. Access the filters by clicking on the "Choose" button.

   • Remove This filter removes one or more attributes from the current dataset.
     1. Select "Remove" from "weka->filters->unsupervised->attribute"
     2. Click on "Remove" to access the Weka window with the parameters for this filter
     3. Click on "More" to learn more about this filter
     4. Use say "1-2" in the "AttributeIndices"
     5. Click "OK"
     6. Click "Apply" on the main Weka Window
     7. Describe how the dataset changed

   • Resample This filter selects a random sample from the data instances in the current dataset.
     1. Select "Resample" from "weka->filters->unsupervised->instance"
     2. Click on "Resample" to access the Weka window with the parameters for this filter
     3. Click on "More" to learn more about this filter
     4. Use say "20.0" in the "SampleSizePercent" to select a subsample of 20% of the current instances.
     5. Click "OK"
     6. Click "Apply" on the main Weka Window
     7. Describe how the dataset changed

7. If time permits, discuss how to convert a dataset to ARFF (attribute relation file format).

• Data sampling
• Attribute discretization
• Missing values (deletion and imputation)
• Dimensionality Reduction:
  • Attribute/feature selection:
    Correlation based Feature Selection (CFS), ReliefF, ...
  • Feature Extraction:
    Singular Value Decomposition (SVD), Principal Components Analysis (PCA), Fourier transform, ...

• ** Python: Data Preprocessing entry on Prof. Ruiz's Miscellaneous Notes on Python

• ** Tan's, Steinbach's, and Kumar's textbook slides - Chapter 2.

• Witten's, Frank's and Hall's textbook slides (slides for 4th edition are available, but I like the 3er edition slides below more).
  • ** Chapter 2, includes concepts, instances, attributes, types of attributes.
  • Chapter 3, includes outputs and knowledge representation.
  • ** Chapter 7, includes attribute discretization, sampling, dimensionality reduction, attribute selection, CFS, PCA/SVD, text to attribute vectors.

• Optional: See some preprocessing filters in Weka.

  Note that for Principal Components Analysis on the Weka's Attribute Selection Tab:

  • CenterData=True does the following:
    1. Normalizes (= centers) the attributes (that is, makes each attributes mean = 0 by subtracting the attribute's mean from each attribute value)
    2. Applies PCA to the data's covariance matrix
  • CenterData=False does the following:
    1. Normalizes (= centers) the attributes (that is, makes each attribute's mean = 0)
    2. Standarizes the attributes (that is, makes each attribute's standard deviation = 1)
    3. Applies PCA to the data's correlation matrix

• Singular Value Decomposition (SVD), Principal Components Analysis (PCA)
  • M.W. Berry, Z. Drmac, and E.R. Jessup, "Matrices, Vector Spaces, and Information Retrieval", SIAM Review, Vol.41, No.2, pp.335-362, 1999.
    • pdf version of the paper available from the SIAM website
    • svd_cooking_example.arff
  • Wall, Michael E., Andreas Rechtsteiner, Luis M. Rocha."Singular value decomposition and principal component analysis". in A Practical Approach to Microarray Data Analysis. D.P. Berrar, W. Dubitzky, M. Granzow, eds. pp. 91-109, Kluwer: Norwell, MA (2003). LANL LA-UR-02-4001.

• credit_history.arff used in the Decision Trees Example

• Full Contact_Lenses_data_set: Entropy calculations.
• Subset Contact Lenses data set: Decision tree construction (exam solutions)

• World Happiness Dataset
  • world_happiness_dataset.txt
  • world_happiness_with_continents.arff
  • small_world_happiness.arff
• ** Piotr Mardziel's numeric predictions over small_world_happiness

• Goodfellow, Bengio and Courville, "Deep Learning". Online version is available at http://www.deeplearningbook.org/
• ** Michael Nielsen, "Neural Networks and Deep Learning", Determination Press, 2015. Neuralnetworksanddeeplearning.com

• Yann LeCun, Yoshua Bengio, Geoffrey Hinton. "Deep learning". Nature 521, 436-444 (28 May 2015) doi:10.1038/nature14539.

• ** Deep Learning - The Past, Present and Future of Artificial Intelligence" by Lukas Masuch.
• ** an introduction to: Deep Learning by Prof. David Wolfe Corne.
• ** Google's ML Practicum: Image Classification Introducing Convolutional Neural Networks.
• deeplearning.net tutorials. Among them:
  • Tutorial from Y. Bengio's LISA lab in Montreal.

• The following videos, among others, are available at deeplearning.net tutorials:
  • "Deep Learning" by Ruslan Salakhutdinov from the collection of Deep Learning Summer School, Montreal 2015
  • "Recent developments on Deep Learning" Geoffrey Hinton's GoogleTech Talk, March 2010.

• "Computer, reply to this email" - what it does
• "Computer, reply to this email" - how it works

• Piotr Mardziel's classification rules over credit_history
  • credit_history.arff

• Solutions to CS4445 B term 2012 Exam 2
• Ken Loomis' Solutions CS4445 B term 2012 HW4

• Quinlan's New Generation Computing 1995 paper
• Quinlan's Machine Learning Journal 1990 paper
• mystery1.arff
• mystery2.arff
• mystery3.arff
• Ackermann Function
• Foil's input: ackermann.d
• What if we run ackermann.arff using Prism?
  • Prism rules for ackermann.arff
• Foil's output for ackermann.d
• Foil's input: member.d
• Foil's input: member.explain *** useful description of foil's input file format
• Foil's input: ncm.d
• Foil's input: qs44.d
• Foil's input: sort.d
• =README=
• MANUAL

• credit_history.arff used in the handouts below
• Prof. Ruiz's Naive Bayes Handout
• Prof. Ruiz's Bayesian Nets Handout

• Expectation Maximization (EM): Slides 150-154.

• Probability and Bayesian Learning: Slides 125-130;
• Gibbs Classifier: slide 144;
• Naive Bayes: Slides 145-156;
• Bayesian Nets: Slides 158-167;
• Expectation Maximization (EM): Slides 168-174.

• Expectation Maximization (EM) and Gibbs Sampling

• diabetes_no_attribute_names.dat
• naive_bayes_example_diabetis.m

• G. F. Cooper & E. Herskovits. "A Bayesian Method for the Induction of Probabilistic Networks from Data".
• For an illustration of the K2 algorithm:
  • see Prof. Ruiz's K2 handout.
  • over the toy dataset in that paper.

1. Introduction
2. Maximal Margin Classifier
3. The Kernel Trick

1. Support vector machines and machine learning on documents

1. Gene Finding and HMMs

1. ** Ydo Wexler & Dan Geiger's Markov Chain Tutorial.
2. ** Hidden Markov Models (HMMs) Tutorial.

1. ** Prof. Ruiz's example of Viterbi's, Forward, and Backward algorithms
2. hmmgenerate_fair_loaded_coins_HMMs_tutorial_example.m using Matlab
3. hmmgenerate_pepsi_coke_HMMs_tutorial_example.m using Matlab

1. HMM Introduction
2. Computing Likelihood
3. HMM Training
4. HMM Applications

1. CpG islands, Markov Chains, HMMs
2. Forward and backward evaluation, HMM training

1. Markov models and Hidden Markov models
2. Estimating and using Hidden Markov models
3. Using HMMs for genefinding: GENSCAN and related models
4. Matlab Demos: Markov and HMM models

• credit_history.arff

• K-Means Clustering using Matlab
• Hierarchical Clustering using Matlab
• EM Clustering using Matlab and Weka

• Look for EM in Bayesian Models

• "Web Mining - Accomplishments & Future Directions." by Jaideep Srivastava, Prasanna Desikan, and.Vipin Kumar.
• ** Jaideep Srivastava's Web Mining Tutorial.

• ** COMAD 2010 tutorial slides on IR and text mining in bioinformatics .

• ** Christopher D. Manning, Prabhakar Raghavan & Hinrich Schütze. "Introduction to Information Retrieval". Cambridge University Press. 2008.
  Use this online textbook as a reference.

• M.W. Berry, Z. Drmac, and E.R. Jessup, "Matrices, Vector Spaces, and Information Retrieval", SIAM Review, Vol.41, No.2, pp.335-362, 1999.

• StringToWordVector filter
• Document classification

• Rebholz-Schuhmann D., Oellrich A., Hoehndorf R. "Text-mining solutions for biomedical research: enabling integrative biology". Nature Reviews Genetics. 2012 Dec;13(12):829-39. doi: 10.1038/nrg3337. Epub 2012 Nov 14.

• Ross D. King et al. "The Automation of Science". Science 324, 85 (2009); DOI: 10.1126/science.1165620

• 2007 paper on progress in biomedical text mining

• 2005 survey of biomedical text mining

• Max Planck Tubingen slides on text mining for bioinformatics

• 2003 short paper on GENIA bio-text mining corpus

• My Bioinformatics and Computational Biology Resources and References.

• Tan's, Steinbach's, and Kumar's textbook slides - Chapter 6.