AIRG Topics - Spring 2006

Our group meets on Thursdays at 11:00 a.m., FL 246.

Dates and topics for this semester are as follows:

Jan 19

AIRG Organizational Meeting (Coordinator: DCB)

Jan 26

Mingyu Feng

"Addressing the Testing Challenge with a Web-Based E-Assessment System that Tutors as it Assesses"

Secondary teachers across the country are being asked to use formative assessment data to inform their classroom instruction. At the same time, critics of No Child Left Behind are calling the bill "No Child Left Untested" emphasizing the negative side of assessment, in that every hour spent assessing students is an hour lost from instruction. Or does it have to be? What if we better integrated assessment into the classroom, and we allowed students to learn during the test? Maybe we could even provide tutoring on the steps of solving problems. Our hypothesis is that we can achieve more accurate assessment by not only using data on whether students get test items right or wrong, but by also using data on the effort required for students to learn how to solve a test item. We provide evidence for this hypothesis using data collected with our E-Assistment system by more than 600 students over the course of the 2004-2005 school year. We also show that we can track student knowledge over time using modern longitudinal data analysis techniques. In a separate paper at this same conference track, we report on the Assistment system.s architecture and scalability, while this paper is focused on how we can reliably assess student learning.
{Paper accepted at WWW'06}

Feb 2

Video

Common Sense: Minsky & McCarthy

Feb 9

Leena Razzaq

"Scaffolding vs. Hints in the Assistment System"

Razzaq et al, (2005) reported that the Assistment system was causing students to learn at the computer but we were not sure if that was simply due to students getting practice or more due to the "intelligent tutoring" that we created and force students to do if they got an item wrong. Our survey indicated that some students found being forced to do scaffolding sometimes frustrating. We were not sure if all of the time we invested into these "fancy" scaffolding questions was worth it. We conducted a simple experiment to see if students learned on a set of 4 items if they were given the scaffolds that tried to ASK for information compared with being given hints that tried to TELL them the same information. Our results show that students that were given the scaffolds performed better with an effect size of 0.3.

Feb 16

Academic Advising Appointment Day: No meeting

Feb 23

Kevin Kardian

"Knowledge Engineering for Intelligent Tutoring Systems: Using machine learning assistance to help humans tag questions to skills based upon the words in the questions"

Building a mapping between items and their related knowledge components, while difficult and time consuming, is central to the task of developing affective intelligent tutoring systems. Improving performance on this task by creating a semi-automatic skill encoding system would facilitate the development of such systems. The goal of this project is to explore techniques involved in text classification to the end of improving the time required to correctly tag items with their associated skills.

Mar 2

Aparna Varde

PhD Progress Report

"DesRept: Designing Domain-Specific Cluster Representatives for Classification"

Experimental results in scientific domains such as Materials Science are often plotted as 2-dimensional graphs of process variables. Performing actual experiments consumes significant time and resources motivating the need for computational estimation. We have proposed an estimation approach, AutoDomainMine, based on integrating clustering and classification. In this approach, graphs from existing experiments are clustered and the clustering criteria are identified using decision tree classifiers. The clusters of graphs and the decision tree paths leading to them are used to build a representative pair of input conditions and graph per cluster. The representative pairs form the basis for classifying new experiments, hence estimating their "graphical results" given their "input conditions", and vice versa. However, we have found that randomly picking a set of conditions and a graph for each cluster as its representative pair is not sufficient since that does not capture all the relevant information in the cluster. This is confirmed by the domain experts. Hence, in this talk, we now propose a methodology called DesRept to design a representative pair of input conditions and graph per cluster incorporating domain knowledge. In DesRept, three candidate representatives each of conditions / graphs are constructed for a given cluster, which we refer to as the nearest, summarized and combined representatives. These representatives are compared using an encoding analogous to MDL that considers the complexity of the representative and its distance from other items in the cluster. The complexity of each representative refers to its visual interpretability while the distance denotes the information loss due to it. The representative with the lowest encoding is considered to be the best for that cluster. We propose to select this as the designed representative of the cluster since it is a good balance between interpretabilty and information loss. DesRept is evaluated in the domain of Heat Treating of Materials that motivated this research. The evaluation is done by comparing the designed representatives with the random ones using the same encoding. Upon performing tests over several data sets using different parameters, it is found that the designed representatives are distinctly better than the random ones. Thus the designed representatives from DesRept are used in the AutoDomainMine system.

Mar 9

Undergrad term break

Mar 16

*Postponed*

Zach Pardos

"Using Bayes Nets to Model Student Knowledge (and later Learning)"

I will present the work I have done using MATLAB to train Bayes Nets models that focus on modeling student performance inside the ASSISTment system. We have a dataset of 600 student who have done about 100 math questions each over the course of a year. I will present results on trying to predict their real MCAS state test performance, taking advantage of with which skills each item are coded. I will sketch briefly my plan to incorporate models of student learning.

Mar 23

Genetic Programming: The Movie, by John R. Koza and James P. Rice, 1992

This video accompanies the book Genetic Programming: On the Programming of Computers by Means of Natural Selection. It provides a general introduction to genetic programming and a visualization of actual computer runs for many of the problems discussed in the book. These problems include symbolic regression, the intertwined spirals, the artificial ant, the truck backer upper, broom balancing, wall following, box moving, the discrete pursuer-evader game, the differential pursuer-evader game, inverse kinematics for controlling a robot arm, emergent collecting behavior, emergent central place foraging, the integer randomizer, the one-dimensional cellular automaton randomizer, the two-dimensional cellular automaton randomizer, task prioritization (Pac Man), programmatic image compression, solving numeric equations for a numeric root, optimization of lizard foraging, Boolean function learning for the 11-multiplexer, co-evolution of game-playing strategies, and hierarchical automatic function definition as applied to learning the Boolean even-11-parity function.

Mar 30

Greg Milette

MS Thesis presentation

"Analogical Matching Using Environment-Centric and Device-Centric Representations of Function"

Advisor: Dave Brown

This AI in Design research focuses on knowledge representation and analogical reasoning. We performed experiments to determine some of the benefits of using environment-centric (EC) vs. device-centric (DC) knowledge representations. We used the Structure Mapping Engine for matching, and show the effect on quality and quantity of analogical matches when the knowledge representation is varied.

Apr 6

No Meeting

Apr 13

Kevin Menard

MS Thesis Presentation

"Evaluating User Feedback Systems"

Advisors: Mark Claypool & Dave Brown

The increasing reliance of people on computers for daily tasks has resulted in a vast number of digital documents. Search engines are now quickly becoming the only practical way to navigate through this sea of information. Traditionally, search engine results are based upon a mathematical formula of document relevance to a search phrase. Often, however, what a user deems to be relevant and what a search engine computes as relevant are not one in the same. User feedback regarding the utility of a search result can be collected in order to refine query results. The most straightforward way of collecting user feedback is to add a graphical user interface component to the search interface that asks the user how much he or she liked the search result. However, whether the feedback mechanism requires users to give feedback or simply asks them too may affect both the quality and quantity of collected data. My work focused on the collection of explicit user feedback in both mandatory (a user must give feedback) and voluntary (a user may give feedback) scenarios. The collected data was used to train a decision tree classifier that provided user satisfaction values as a function of implicit user behavior and a set of search terms. The results of my study indicate that a more accurate classifier can be built from explicit data collected in a voluntary scenario. Given a limited search domain, the classification accuracy can be further improved.

Apr 20

Zach Pardos

"Using Bayes Nets to Model Student Knowledge (and later Learning)"

I will present the work I have done using MATLAB to train Bayes Nets models that focus on modeling student performance inside the ASSISTment system. We have a dataset of 600 student who have done about 100 math questions each over the course of a year. I will present results on trying to predict their real MCAS state test performance, taking advantage of with which skills each item is coded. I will sketch briefly my plan to incorporate models of student learning.

Apr 27

No meeting