Kelly et al’s Archive

This web page contains the material used for a scientific study so that others can see what we did, and attempt to replicate it. I am trying to use http://www.webcitation.org to preserve this information in an archival quality manner. WebCite will store all small web pages so even if Heffernan’s post of these links on his web page changes, other researchers should be able to get the materials. The only data we could not store with WebCite were the YouTube videos and the running of the homework in ASSISTments. This page itself is achieved at http://www.webcitation.org/6E6lv54G8 buts easier to browse using the original page http://web.cs.wpi.edu/~nth/PublicScienceArchive/Kelly.htm.

Materials (Pretest, Posttest1 and Posttest2, plus screenshots and classroom videos):

1. Pre-Test: (Problem Set 35482): http://www.webcitation.org/6CoNyW7ad

2. The homework questions were these: http://www.webcitation.org/6DziiZN3k

a. Here is some detailed analysis that goes across the two conditions regarding the triplets and where they showed up in all videos. http://www.webcitation.org/6E03MzG3a Browse here.

b. Experimental Condition (Referred to as WBH for Web Based Homework.)

i. Homework in the Experimental Condition (Problem Set number 35511): http://www.assistments.org/public_preview/link/dHlwZT1zZXF1ZW5jZSZpZD0zNTUxMQ== (you need to make sure the include the two equal signs at the end. If your browser leaves them off, copy the ULR by hand and make sure you get the two equals signs. (You can create an ASSISTment account and assign problem set 35511to run this problem set.)

ii. Screenshot: Here is what the screen shot looked like in the experimental condition http://www.webcitation.org/6Dzgr4Pss (You can’t see that when the teacher hovers his her mouse over a column header they can see you question text. Note how the fourth question has 27% correct and that 56% of student that got that question wrong said the same wrong answer of “1/9^10”. The teacher goes over this in the teacher video below.

iii. Video: Videos of the teacher going over the homework (Experimental Condition): http://www.youtube.com/watch?feature=player_embedded&v=Jb6Szy4fZ2w

c. Control Condition represents traditional paper and pencil (Referred to as TH for Tradition Homework) but actually done in “test” mode in ASSISTments to time students.

i. Homework in the Control Condition (Problem Set number 35699): http://www.assistments.org/public_preview/link/dHlwZT1zZXF1ZW5jZSZpZD0zNTY5OQ==

ii. Screenshot: If the teacher had logged in to look at the item report for the control conditions, this is what it looked liked, but for this experiment the teacher did not look at this reports until after the experiment was over. http://www.webcitation.org/6DzgwBUdY

iii. Video: Videos of the teacher going over the homework (Control Condition): http://www.youtube.com/watch?feature=player_embedded&v=tBhcuCnKVCY

3. PostTest1 (Problem Set 35483) http://www.webcitation.org/6DzfG1wer

4. PostTest2 (Problem Set 35484): http://www.webcitation.org/6CoO51gb4

Results:

5. Survey results student did that are anonymized and stored here. The comments students made are interesting to read. http://www.webcitation.org/6DzciCGXm

6. Here is an excel file holding all the data (anonymized) so others could reanalyze our results with different statistical methods. http://www.webcitation.org/6DzhQ1Agl

7. We relied on this CEMs spreadsheet to computer the Hedges’ Corrected effect sizes and 95% confidence intervals http://www.webcitation.org/6DzhXgqjJ

The submitted paper is at http://web.cs.wpi.edu/~nth/pubs_and_grants/papers/2013/AIED2013/Kelly2013AIEDSub.pdf

Kelly, Y., Heffernan, N., Heffernan, C., Goldman, S., Pellegrino, G. & Soffer, D. (submitted). Estimating the Effect of Web-Based Homework. The Artificial Intelligence in Education Conference. Memphis. 2013.

Kim Kelly(2012)