Assessment of Tools for Virtual Teaching
by
Karen A. Lemone
Associate Professor
Computer Science Department
Worcester Polytechnic Institute
Worcester, MA 01690
+1 508 831 5529
kal@cs.wpi.edu
Abstract
Virtual teaching via the Web is becoming commonplace. Tools to better enable
this activity sre beginning to appear. However, little formal assessment
has been done to determine their effectiveness nor the effectiveness of
such distance learning. In this paper, we describe a set of tools which aid
both the instructor and the student as well as assessment procedures
for evaluating their use.
1. Introduction
Teaching courses via Web materials has new teaching issues plus old issues
in a new setting:
- Just as in traditional courses, TA's and other assistants are needed.
For Web-based courses, however, they are needed for maintainance of pages,
answering student questions - asynchronously via email and synchronously
by holding "office hours" in Chat Rooms.
- While routine homeworks can be graded, recorded and responded to
automatically, good software tools to enable this are just appearing.
- When instructors teach a course for the
second (third, fourth, ...) time, they reorganize existing material to make
it appropriate for the current class. In traditional mode, this may include
adding and deleting material, creating new projects, quizzes and assignments,
refocusing for a different audience, etc. Software to facilitate these tasks
needs to be developed and tested.
- The Web provides poor facilities for searching and navigating.
Supplemental tools are needed.
ReCourse [ Lemone, 1996], which has been evolving over the last three years, is a Web
Retargetable Course Generation System whose purpose is to facilitate both distance and on-campus learning via the World Wide Web. By "retargetable"
we mean the process of changing the Web course to "target" it for a different
term or audience. It is a system which will work with any Web course; it is
not tied to a particular Web course. The concept is tied to automatic
document assembly, which could be used in other areas such as
Network Publishing.
Its features include:
The ability to retarget a Web course for different levels of students. A user-friendly
editor allows instructors to add appropriate tags to HTML documents. Students then see only the parts of the pages appropriate for their level.
Multiuser
chat rooms to facilitate synchronous student, instructor, and TA
communication.
A secure grading system allowing instructors to record grades and students to view their own grades.
Bookkeeping Tools such as a Hypertext Link Check to ensure that all internal and external hypertext references are valid, and Content Update tools to allow global updating of course pages (e.g., changing the term and date headers, course icons etc.)
A Map Generator to create a semi-static site map of the pages to allow students a birds-eye view of where they are in the course pages. This tool is run periodically by either the instructor or TA's when changes have been made in the
organization of the course pages.
A Quiz Feedback system.
A Search engine so students can locate material for homeworks etc.
2. Instructional Model
People have been teaching courses via the Web for a number
of years now. Sometimes the Web is used as a supplement to the
class. Sometimes it is where the class takes place.
We have experimented with a number of
models and
instructional designs and have learned and are still learning about the
impact on student learning and faculty productivity of these models.
2.1 Instructional Design
ReCourse is a Web-based system used in conjunction with Web
course pages. It presumes course pages exist in a directory, and that
there is a "root node" (home page); other pages are connected as links in the
typical web-like architecture. Future enhancements will faciltitate this
creation. Currently, it is presumed that such a directory of web pages exists.
A typical course would have a number of modules representing the major
topics in the course. Links also exist to the course information - email and
phones of the instructor, TA and graders, Syllabus, Class list - with references to
their home pages (if any) and their email addresses - Project decription
(if any), and grading.
2.2 Educational Technology
Although the Web courses may be used within the classroom structure,
Web ReCourse is really a distance learning model. Having taught this way
for three summers, we have developed and incorporated techniques to
facilitate distance
learning: multiple (Web) references and weekly homeworks
for reinforcement of the material, personalized responses when homework
is submitted, and "presence" (asychronously via email, synchronously via
Chat Rooms). In addition, the tools include automatic
grading feedback on homework, a search engine and birds-eye views of pages
so that students can see where they are in the material and find other
information more quickly.
2.3. Comparison with Other
Instructional Models
Non Web-based distance learning models have relied on videotapes and
broadcasts. While some Web courses have been taught synchronously
via White Boards, etc., the technology just isn't sufficient yet. Our model
is primarily asynchronous, allowing both the instructor and students
to work at their own place, rate, and time.
Most Web-based courses are created and maintained by the instructors, perhaps with TA help. Few systems exist to aid the teaching of Web courses. WebCT [Goldberg 96, http://homebrew.cs.ubc.ca:8080/] comes the closest to ReCourse, but it lacks the "retargeting" facilities: when a course is retaught, it needs to be changed, updated, etc. Web courses take a phenomenal amount of time to develop, update and maintain. Tools to reuse material are needed. We know of no other system that addresses this retargeting issue.
During the preceding three summers, we have collected statistics on time spent
by both students and the instructor. This is the first year that all the tools
will be available.
Productivity should improve for the instructor and students due to:
TA help in chat rooms and email. We spent hours each week, responding
to email in the past. Sometimes, we could not respond in a timely manner.
Support personnel are needed for distance learning in many of the same ways
that they are needed for traditional classes.
Automatic grading of weekly homeworks. We use routine assignments
to encourage reading and assimilating of the course material. In the
past, we have graded them ourselves and sent students feedback and their scores via email.
Again, this has taken a few hours/week. The automatic test system
will ease this.
The Bookkeeping Tools and Retargeting tools enable
the instructor to create the next version of the class in far less time
than we presently spend.
Instantaneous feedback to students on their homework.
Personal attention via the Chat rooms. This has been used in a WPI
Compiler
Design class. Students found it extremely helpful. The instructor
and TA also found it convenient to hold "office hours" from home.
Search and Map tools to make perusing the course easier.
Automatic and secure access to student grades (for students and the instructor.)
3. Assessment Plan
We have been funded by the Davis Educational Foundation to develop
and perform statistically significant assessments on these classes. The
Pretest and Posttest described here are the initial results of this work.
3.1 Procedures and Instruments to Measure Effectiveness
We have been using student questionnaires for the last 3 years. There is a preliminary questionnaire, and a post questionnaire for each course.
One term, students filled out weekly assessments. Interestingly, students have always
filled out these electronic Web forms even when they run a week or two
behind. We've never gotten anywhere near this response with paper questionnaires.
3.2 Description of Control Group and Comparison Tools
We will be assessing the effectiveness of ReCourse in the summer versions
of two classes: Electronic Documents and Network Publishing. The
Network Publishing group are less technical, more writing
and publishing-oriented (in theory). The Electronic Documents group are
Computer Science or Computer Engineering majors (or those with strong
computer backgrounds.) We will be comparing these groups, not with each
other, but with information we have been and will be gathering in previous
and future
versions of the course. We will be comparing issues such as
(1) time spent, (2) knowledge and skills gained, and (3) satisfaction.
3.3 Pre/Post Analysis
For the preliminary questionnaire, we ask questions about their background
and interests. This year are adding many technical questions -
information and skills taught in the course - and then ask these
questions again on the post questionnaire. We are adding questions
concerning attitude also. For example, "How important is a usability test
of your project"? (Usability testing is a hard sell in these courses.) We
also plan to do brief weekly assessments. Some questions will be:
"how difficult/interesting/clear/relevant etc. was the week's material"?
"How much time was spent on the class work?" "on the project"?
We can compare these to the times reported two years ago.
Because we ask some of the same questions from year to year on the weekly
homeworks, we can learn whether students are answering more questions
correctly because of the tools.
We will also use the WPI standard course evaluation form (See Appendix B)
(The first 14 questions indicate an overall measure of satisfaction, and
the very last question indicates
self-perception of learning.) We will compare these with previous versions of the courses to analyze differences.
As the instructor, I will also keep an electronic journal with the
amount of time spent, any frustrations etc.
4. Outcomes Statement
4.1 Measurable Quality Outcomes Resulting from Innovation
The easiest statistic to measure will be times. In the past,
both students and the instructor have spent unreasonably large amounts
of time. A reasonable amount of effort time-wise is one of the hoped
for outcomes of the ReCourse system.
Tabulating the correct answers to the technical questions in the
post test (but subtracting off for those who knew a particular topic
as evidenced in the pre-test) will give feedback on what topics are
being learned and which ones need to be be presented in different ways.
Class satisfaction has been high in the past. Students seem to like taking
a course (mostly) on their own in the summer. Although not as objective
as times and correct answers to a question, it can still be measured,
at least qualitatively, and reported on.
Comparison of the student's desired outcome ("What do you hope to learn in this course?") described on the pre-test with the actual outcome ("Did you learn (less than/more than/ etc. ) what you hoped to learn") on the post-test, is an important measurable. (We email back right away when a desired goal is unrealistic
for the course.)
These are important outcomes. If classes are to be taught virtually
(at least part of the time), students should be learning, and should
be satisfied with the way they are learning.
5. Description of Pretest and Posttest
The assessment is intended to measure technical knowledge, attitude and
satisfaction and gather statistics about the student/learner. The method
chosen is to alternate these. Lines separate the sections of the tests.
Thus, the first set of questions gather statistics. This is followed by the
first set of technical questions. This alternation will continue. the structure
is:
- Questions about students background etc.
- Technical questions related to Module 1
- Questions about student background etc.
- Technical questions related to Module 2
- Questions about student behaviors
- Technical questions related to Module 3
- Questions about student attitudes
- Technical questions related to Module 4
- Questions about student attitudes
- Technical questions related to Module 5
- Questions about student satisfaction
- Technical questions related to Module 6
Because there are 100 short technical
questions, we believe that students will not remember the initial
questions when they take the Postest. The exam will be given on paper
at the first class meeting, collected and not returned.
The technical questions are intended to assess students knowledge
of the material, analytical skills, problem solving skills, critical
thinking skills,
and technological skills. Other aspects of the course will assess
their inquiry and research skills (the project) and their presentation
skills (each student presents their project at the second and last
class meeting.)
Appendix A shows the pretest for a course in
Electronic Documents. Both this and the posttest
(an edited version of the pretest) will be administered in the course
which begins in June 1997.
6. Conclusions
ReCourse is a software tool that facilitates and enhances Web courses,
making it easier and more effective for both students and instructors.
The design of ReCourse has been described in Retargetable Course Generation, A Methodology for Reusability , in Proceedings of Workshop on Architectures for Intelligent Tutoring Systems, ITS '96, Montreal, Canada, June 1996.
Now that the tools described here have been implemented (some have
been used), it is time to assess both them and teaching and learning
via the Web.
Appendix A - The Pretest
I1. Your name:___________________________________________
I2. Your email address:__________________________________
I3. What is your field or major?__________________________
I4. Do you have a Web home page? Yes No
I5. Have you used HTML? Yes No
I6. If so, how much?__________________________________________
I7. What World Wide Browsers are you familiar with?
Technical Questions
I8. Do you know Perl? Yes No
I9. Do you know Java? Yes No
I10. Do you know JavaScript? Yes No
I11. Why are you taking this class?
I12. What do you hope to learn in this class?
I13. Will you use an Internet Provider during this course? Yes No
Technical Questions
B1. How many hours a week do you plan to spend on the course?
(a) Ten or fewer
(b) Eleven to fifteen
(c) Sixteen to twenty
(d) More than 20
B2. How many different day/week do you think you will work on the
course?
(a) One
(b) Two
(c) Three
(d) Four or more
B3. What computers will you use?
(Circle all that apply)
(a) WPI's at WPI
(b) WPI's from work or home
(c) PC at home
(d) Mac at home
(e) PC at work
(f) Mac at work
(g) Unix machine at work
(h) Other (specify)_________
B4. How do you plan to read the material on the Web?
(a) On the screen
(b) Print it all out
(c) Partially on screen; partially on paper
B5. How many hours a week do you currently spend "surfing" the Web?
(a) Zero
(b) One to five
(c) Five to ten
(d) Ten or more
Technical Questions
A1. Do you like the idea of taking this courses almost entirely
online? Yes No Not sure
A2. Do you think you will visit all the pages related to this course?
Yes No Not sure
A3. Would you have preferred to take this class the traditional
in-class way? Yes No
A4. Are you concerned that accessing the Web may be slow?
Yes No Not sure
A5. Do you think this course could be done with no meetings at all?
Yes No Not sure
Technical Questions
A6. Is the fact that you don't have to come to a class important to you? Yes No
A7. Is the fact that you can do this course "in your own time"
important to you? Yes No
A8. Do you think online "labs" will work? Yes No Not sure
A9. Do you plan to use the online Chat Room?Yes No Not sure
A10. Do you think you'll be able to do a project online? Yes No Not sure
Technical Questions
S1. Are the course objectives clear to you? Yes No
S2. Do you think the course is well organized? Yes No
S3. Do you think you'll be able to communicate with the instructor and
PLA's? Yes No Not sure
S4. Do you think the course material is challenging? Yes No Not sure
S5. Do you think the course material is interesting? Yes No Not sure
S6. Do you think you'll take other courses this way? Yes No Not sure
S7. Has the instructor been helpful so far? Yes No Not sure
S8. Do you think you'll be able to apply the material and skills from
this course to your job? Yes No Not sure
S9. Do you think you will learn a lot in this course? Yes No Not sure
S10. Do you think the evaluations (homeworks and this assessment)
measure your knowledge of the material well? Yes No Not sure
Technical Questions
Appendix B - WPI Course Evaluations
Side 1
Side 1
