Experiences in Virtual Teaching
Abstract
Virtual teaching via the Web is becoming commonplace. Tools to better enable
this activity are beginning to appear. However, little formal assessment
has been done to determine their effectiveness nor the effectiveness of
such distance learning. In this paper, we describe a set of tools which aid
both the instructor and the student as well as assessment procedures
for evaluating their use.
1. Introduction
Teaching courses via Web materials has new teaching issues plus old issues
in a new setting:
- Just as in traditional courses, TA's and other assistants are needed.
Here, however, they are needed for maintainance of pages,
answering student questions - asynchronously via email and synchronously
by holding "office hours" in Chat Rooms.
- While routine homeworks can be graded, recorded and responded to
automatically, good software tools to enable this are just being developed.
- When instructors teach a course for the
second (third, fourth, ...) time, they reorganize existing material to make
it appropriate for the current class. In traditional mode, this may include
adding and deleting material, creating new projects, quizzes and assignments,
refocusing for a different audience, etc. Software to facilitate these tasks
needs to be developed and tested.
- The Web provides poor facilities for searching and navigating.
Supplemental tools are needed.
ReCourse [ Lemone, 1996], which has been evolving over the last three years, is a Web
Retargetable Course Generation System whose purpose is to facilitate both distance and on-campus learning via the World Wide Web. By "retargetable"
we mean the process of changing the Web course to "target" it for a different
term or audience. It is a system which will work with any Web course; it is
not tied to a particular Web course.
Its features include:
Ability to retarget a Web course for different levels of students. A user-friendly
editor allows instructors to add appropriate tags to HTML documents. Students then see only the parts of the pages appropriate for their level.
Multiuser
chat rooms to facilitate synchronous student, instructor, and TA
communication.
A secure grading system allowing instructors to record grades and students to view their own grades.
Bookkeeping Tools such as a Hypertext Link Check to ensure that all internal and external hypertext references are valid, and Content Update tools to allow global updating of course pages (e.g., changing the term and date headers, course icons etc.)
A Map Generator to create a semi-static site map of the pages to allow students a birds-eye view of where they are in the course pages. This tool is run periodically by either the instructor or TA's when changes have been made in the
organization of the course pages.
A Quiz Feedback system.
2. Instructional Model
People have been teaching courses via the Web for a number
of years now. Sometimes the Web is used as a supplement to the
class. Sometimes it is where the class takes place.
We have experimented with a number of
models and
instructional designs and have learned and are still learning about the
impact on student learning and faculty productivity of these models.
2.1 Instructional Design
ReCourse is a Web-based system used in conjunction with Web
course pages. It presumes course pages exist in a directory, and that
there is a "root node" (home page); other pages are connected as links in the
typical web-like architecture. Future enhancements can faciltitate this
creation. Currently, it is presumed that such a directory of web pages exists.
A typical course would have a number of modules representing the major
topics in the course. Links also exist to the course information - email and
phones of the instructor, TA and graders, Syllabus, Class list - with references to
their home pages (if any) and their email addresses - Project decription
(if any), and grading.
2.2 Educational Technology
Although the Web courses may be used within the classroom structure,
Web ReCourse is really a distance learning model. Having taught this way
for three summers we have developed and incorporated techniques to
facilitate distance
learning: multiple (Web) references and weekly homeworks
for reinforcement of the material, personalized responses when homework
is submitted, and "presence" (asychronously via email, synchronously via
Chat Rooms). In addition, the tools include automatic
grading feedback on homework and birds-eye views of pages so that students
can see where they are in the material and find other information more
quickly.
2.3. Comparison with Other
Instructional Models
Non Web-based distance learning models have relied on videotapes and
broadcasts. While some Web courses have been taught synchronously
via White Boards, etc., the technology just isn't sufficient yet. Our model
is primarily asynchronous, allowing both the instructor and students
to work at their own place, rate, and time.
Most Web-based courses are created and maintained by the instructors, perhaps with TA help. Few systems exist to aid the teaching of Web courses. WebCT [Goldberg 96, http://homebrew.cs.ubc.ca:8080/] comes the closest to ReCourse, but it lacks the "retargeting" facilities: when a course is retaught, it needs to be changed, updated, etc. Web courses take a phenomenal amount of time to develop, update and maintain. Tools to reuse material are needed. We know of no other system that addresses this retargeting issue.
During the preceding three summers, we have collected statistics on time spent
by both students and the instructor. This is the first year that all the tools
will be available.
Productivity should improve for the instructor and students due to:
TA help in chat rooms and email. We spent hours each week, responding
to email in the past. Sometimes, we could not respond in a timely manner.
Support personnel are needed for distance learning in many of the same ways
that they are needed for traditional classes.
Automatic grading of weekly homeworks. We use routine assignments
to encourage reading and assimilating of the course material. In the
past, we have graded them ourselves and sent students feedback and their scores via email.
Again, this has taken a few hours/week. The automatic test system
will ease this.
The Bookkeeping Tools and Retargeting tools enable
the instructor to create the next version of the class in far less time
than we presently spend.
Instantaneous feedback to students on their homework.
Personal attention via the Chat rooms. This has been used in a WPI
Compiler
Design class. Students found it extremely helpful. The instructor
and TA also found it convenient to hold "office hours" from home.
Search and Map tools to make perusing the course easier.
Automatic and secure access to student grades (for students and the instructor.)
3. Assessment Plan
We have been funded by the Davis Educational Foundation to develop
and perform statistically significant assessments on these classes.
3.1 Procedures and Instruments to Measure Effectiveness
We have been using student questionnaires for the last 3 years. There is a preliminary questionnaire, and a post questionnaire for each course.
One term, students filled out weekly assessments. Interestingly, students have always
filled out these electronic Web forms even when they run a week or two
behind. We've never gotten anywhere near this response with paper questionnaires.
3.2 Description of Control Group and Comparison Tools
We will be assessing the effectiveness of ReCourse in the summer versions
of two classes: Electronic Documents and Network Publishing. The
Network Publishing group are less technical, more writing
and publishing-oriented (in theory). The Electronic Documents group are
Computer Science or Computer Engineering majors (or those with strong
computer backgrounds.) We will be comparing these groups, not with each
other, but with information we have been and will be gathering in previous
and future
versions of the course. We will be comparing issues such as
(1) time spent, (2) knowledge and skills gained, and (3) satisfaction.
3.3 Pre/Post Analysis
For the preliminary questionnaire, we ask questions about their background
and interests. This year are adding many technical questions -
information and skills taught in the course - and then ask these
questions again on the post questionnaire. We are adding questions
concerning attitude also. For example, "How important is a usability test
of your project"? (Usability testing is a hard sell in these courses.) We
also plan to do brief weekly assessments. Some questions will be:
"how difficult/interesting/clear/relevant etc. was the week's material"?
"How much time was spent on the class work?" "on the project"?
We can compare these to the times reported two years ago.
Because we ask some of the same questions from year to year on the weekly
homeworks, we can learn whether students are answering more questions
correctly.
We will also use the WPI standard course evaluation form (The first 14 questions indicate an
overall measure of satisfaction, and the very last question indicates
self-perception of learning.) We will compare these with previous versions of the courses
to analyze differences.
4. Outcomes Statement
4.1 Measurable Quality Outcomes Resulting from Innovation
The easiest statistic to measure will be times. In the past,
both students and the instructor have spent unreasonably large amounts
of time. A reasonable amount of effort time-wise is one of the hoped
for outcomes of the ReCourse system.
Tabulating the correct answers to the technical questions in the
post test (but subtracting off for those who knew a particular topic
as evidenced in the pre-test) will give feedback on what topics are
being learned and which ones need to be be presented in different ways.
Class satisfaction has been high in the past. Students seem to like taking
a course (mostly) on their own in the summer. Although not as objective
as times and correct answers to a question, it can still be measured,
at least qualitatively, and reported on.
Comparison of the student's desired outcome ("What do you hope to learn in this course?") described on the pre-test with the actual outcome ("Did you learn (less than/more than/ etc. ) what you hoped to learn") on the post-test, is an important measurable. (We email back right away when a desired goal is unrealistic
for the course.)
These are important outcomes. If classes are to be taught virtually
(at least part of the time), students should be learning, and should
be satisfied with the way they are learning.
5. Conclusions
ReCourse is a software tool that facilitates and enhances Web courses,
making it easier and more effective for both students and instructors.
The design of ReCourse has been described in Retargetable Course Generation, A Methodology for Reusability , in Proceedings of Workshop on Architectures for Intelligent Tutoring Systems, ITS '96, Montreal, Canada, June 1996.
Now that the tools described here have been implemented (some have
been used), it is time to assess both them and teaching and learning
via the Web.
