CS 4536: Programming Languages (C08)

What is Test Jousting?

Test jousting is our internal class testing competition. For each assignment, you will submit test suites and we will run each students' test suite against all students' programs. You will get points for each error you find in a classmate's program; you will lose points for each error someone else finds in yours.

We could just run the jousts after homeworks are submitted. However, to encourage you to think about testing and to give you early feedback on your work, we offer an incentive system: you may submit either tests or code before the deadline. If you submit tests, we'll store them and run them against other students' early submissions. If you submit code, we'll run it against all the valid tests others have submitted so far. Here's the early-testing incentive: you'll accumulate testing points for every early bug you catch, every time you catch it. Submitting a killer test case early could gain you several points. Correspondingly, you'll lose points every time you submit an early program that fails an existing test suite, so you have to do a fair amount of testing on your own, rather than relying on the testing largess of others.

Logistics

Test-jousting will run through a web application. We will set up accounts for each student during the first week of class. You should receive email to your wpi.edu account with an initial password.

Submit/revise a test suite: Cut and paste your test suite from DrScheme into the window provided. We will check that your test suite parses and run it against the reference implementation. If a test fails, you will get email indicating which test failed and the result that our implementation produced on it. None of your tests will be entered into the pool. If all of your tests pass, we will accept your test suite and send it to a TA to approve (just to make sure your file looks like a legitimate test suite). Once the suite is approved, it will be run against all subsequent submissions and you will start accumulating points.

Each test should be an instance of the test or test/exn constructs from the PLAI language levels. Check the DrScheme helpdesk if you need details on these.
Submit a program: Cut and paste your program from DrScheme into the window provided. We will check that your program parses and run it against the current tests in the pool. You will get email with the results of the test.

The testing results will not report the contents of the test cases for which your code failed. This is intentional. You cannot use test jousting to help you figure out how to think about the assignments, and seeing the contents of others' tests would alert you to issues you might not have considered. You can trust that the test suites in the pool are valid, however, because tests are only saved in the pool if they produce the right answer against our reference implementation.

You may submit programs as often as you like, but you will accumulate deductions each time a test suite finds a bug in one of your submissions, so you have an incentive to have confidence in your work before submitting. What's the incentive for taking the risk? You'll lose fewer points for bugs identified during jousting than those identified when we grade your final submission.

The jousting infrastructure requires that your program provide definitions for all functions under test. If an assignment expects you to test functions foo and bar, you will get an error if you submit a program that omits one of them. We will clearly indicate which functions are under jousting test for each assignment.

If you copy your code out of some editor other than DrScheme, omit the three lines at the top (the ones listed as inserted by DrScheme); submit only the actual code or tests.

FAQ

Will I lose points on test cases that fail the reference implementation? No. You only lose points on programs that fail approved test cases.
Are all of my test suites run, or just the most recent one? Just the most recent, so insert the complete test suite each time you want to revise it.
If many students' test suites catch the same bug in my code, how many times will I get penalized for it? You'll lose a jousting point for every test case in every test suite that fails on your program. This is reasonable, because if a test occurs in many test suites, it represents a common scenario that you should have been more likely to think of.
Are we required to submit programs for jousting before the final submissions? No, but you will lose more points for bugs that we find during grading than for those found during jousting. How you weigh the tradeoff is up to you.