Section B: Artificial Neural Networks and Deep Learning
(50 points)
Dataset: For this part of the project, you will use the
OptDigit Dataset available at the
UCI Machine Learning Repository.
- Carefully read the description provided for this dataset and
familiarize yourself with the dataset as much as possible.
- Use the following files:
- optdigits.names
- optdigits.tra: training dataset
- optdigits.tes: test dataset
- Classification using Artificial Neural Networks (ANNs):
Use Matlab functions to construct and train ANNs over optdigits.tra
and then test them over optdigits.tes.
Topology of your Neural Net:
- Layers:
I suggest that you use a 3-layer, feedforward architecture. More
specifically, a net consisting of (1 input layer,) 2 hidden layers,
and 1 output layer. Each node in a layer is connected to each
and everyone of the nodes in the next layer, and no nodes on
the same layer are connected.
- Number of nodes per layers:
The input layer will have an entry for each input attribute.
You will need to determine experimentally
how many nodes to use in each hidden layer
(I recommend to start with relatively small numbers of hidden nodes
and increase the number as needed. Also, it makes sense for the first
hidden layer to contain more nodes than the 2nd hidden layer).
For the output layer, you need to determine if you want to use 10 different
output nodes or just one. Your training the network should be
consistent with this decision.
- However, you can experiment
with other architures in addition to the one suggested here.
Experiments:
- (5 points)
Briefly describe in your report the Matlab functions that you use and
their parameters.
- (5 points)
Explain also how many nodes you use on the output layer, and how
you use the output from the output node(s) to assign a classification
label to a test instance.
- (35 points)
Run at least 10 different experiments varying parameter values.
Show the results of all of your experiments neatly organized on a table
showing parameter values,
number of hidden nodes in each layer,
classification accuracy,
and runtime.
- (5 points)
Pick the experiment that you think produced the best result.
Justify your choice in your report.
Include the confusion matrix for this experiment.
See what misclassifications are most common and
elaborate on your observations.
- Deep Learning:
- Read the following article:
Yann LeCun, Yoshua Bengio, Geoffrey Hinton.
"Deep learning". Nature 521, 436-444 (28 May 2015)
doi:10.1038/nature14539.
- Watch one of the following videos about deep learning
(if you have time try to watch both).
You're not expected to understand all the details, but try to
get from the videos some of the theoretical foundations
of deep learning and some of its applications.
-
"Deep Learning" by Ruslan Salakhutdinov
from the collection of
Deep Learning Summer School, Montreal 2015
-
"Recent developments on Deep Learning"
Geoffrey Hinton's GoogleTech Talk, March 2010.
Links to both videos (and several others) are available at
deeplearning.net tutorials
- Homework Problems:
These homework problems are for you to study this topic.
You do NOT need to submit your solutions.
- Chapter 11 Exercises 1, 2, 3, 6, 12
of the textbook (pp. 311-313).
- Study
convolutional neural networks (section 11.8.3),
autoencoders (section 11.11),
and recurrent networks (section 11.12.2).
You should know the topology (= structure) of these nets and
the procedure to train them.