Slides
``Comparing CS Research Methodologies''

CS Colloquium, Friday 7th November 1997, 11:00 am

Comparing Computer Science Research Methodologies

David C. Brown
Mark Claypool
Stanley M. Selkow

Computer Science Department
Worcester Polytechnic Institute
Worcester, MA 01609, USA.

---

Introduction

• We want to do better research.

• Better research should be obtainable by using better Research Methodology.

• Inspiration:
  • Yoram Reich, ``The study of design research methodology'',
    Journal of Mechanical Design, ASME, vol.117, no. 2(A), pp.211-214, 1995.

---

Overview

1. Definition of ``methodology'': Dave Brown.
2. Consequences of Definition: Dave Brown.
3. The ``Received Scientific Methodology'': Dave Brown.
4. How being in Computer Science might affect one's methodology: Dave Brown.

5. The methodology of the ``Experimentalist'': Mark Claypool.
6. The methodology of the ``Theorist'': Stanley Selkow.

7. The methodologies we use and how we might improve them: Group discussion.

---

Definition

{from Reich}

A methodology is (or attempts to approximate) a compatible collection of:

• assumptions and goals underlying methods,
• the methods, and
• the way the results of carrying out the methods are
  • interpreted and
  • evaluated.

Note that this doesn't quite capture the procedural nature of a methodology
(i.e., as some rational ordering of methods)

---

Consequences of Definition

To discuss a methodology we must know ...

... the assumptions being made

... the goals to be achieved

... what the methods are

... what results each method might generate

... how interpretation of results is carried out

... how (the interpretation of) results are evaluated

To improve one's methodology one needs to ...

... understand why that's the methodology one is using

... make hidden things more explicit

... add missing things

... evaluate how reasonable all methodology ingredients are

e.g. is the evaluation appropriate?

... used better or more appropriate ingredients

e.g., ones that match assumptions and achieve goals.

---

The ``Received Scientific Methodology''

• We tend to want to do what we think scientists are supposed to do:
  1. make observations;
  2. hypothesis formation;
  3. hypothesis testing;
  4. hypothesis evaluation;
  5. hypothesis acceptance or rejection.

• Reich claims that this is impossible to follow and hard to approximate.
• However, most activity can be expressed (at some level of abstraction) as
  PLAN - GENERATE - TEST.

• Whether or not we like this `classical' definition, we tend to use the methodologies of ``close senior peers''.
  i.e., there's a strong social dimension to this.

---

The effect of Computer Science on one's methodology

• The common thread in all subareas of Computer Science is the Computer.
• Research should be about or for the computer:
  • Architecture is about the computer;
  • Software (systems/languages) is about things that run on the computer;
  • Algorithms affect the quality of the software;
  • Theory affects the quality of the Algorithms.

• Hence the computer must play a role of some kind in the methodologies used for each kind of CS research.
  • As part of an assumption
    e.g., a computer model is more appropriate than a physical model (Simulation)
  • As part of a goal
    e.g., to make the computer process a database update quicker (Databases)
  • As part of a method
    e.g., timing a portion of an operating system using the computer's clock (OS)
  • As part of an interpretation
    e.g., presenting data in an organized fashion (Visualization)
  • As part of an evaluation
    e.g., determining the probability of a sent message arriving (Networks)

---

The methodology of the ``Experimentalist''

• Link to Mark's web-based slides
  http://www.cs.wpi.edu/~claypool/exp-method/

---

The methodology of the ``Theorist''

• Link to Stan's web-based slides (.ps)
  http://www.cs.wpi.edu/~sms/method.ps

---

Discussion

• what variations are there in CS Methodologies?

• how can we improve our Methodologies?

• how do we educate Graduate students about these issues?

• avoid meta-meta-... -level discussions
  e.g., a methodology for the improvement of the methodology for the improvement of the methodology ...

• how to continue this discussion?

• how can we get out of here and have lunch?