Creating and Editing Websheets

This manual describes how to write new exercises. If you instead want to learn about installation or usage, see the "about" page.

To get started, we recommend diving in by opening these two links, which demonstrate an exercise called TwoPlusTwo:

Look at the author interface. The most important part of the exercise definition is the Template / Reference solution:
public static void main(String[] args) {
\[
   System.out.print(2+2);
]\
}
In the student interface, the \[...]\ region is blanked-out and has to be filled in. Try making some incorrect and correct submissions.

Moreover, the Template / Reference Solution specifies a correct "reference" solution, which is to fill in that blank with System.out.print(2+2);.

(The grader actually runs the code, so any other solution that behaves the same way is just as good. So System.out.print(4); is correct too.)

Here are the sections of this manual, which describe how to write each component in more detail:

Examples

Looking at existing examples is the most thorough way to understand what's going on, once you're familiar with the main concepts.

You can access a list of all Websheets (login required). There, you can see the definition of about 200 sample exercises in Java and C++.

Short Answer / Multiple Choice

Because we use them occasionally in our courses, Websheets supports a couple of non-coding types of exercises too. Rather than document them, we just give a couple of examples: The rest of the manual talks about coding problems only.

Template / Reference solution

When writing a websheet, normally you will start by writing a correct solution to the problem. (For Java, note that you can either write in public class ExerciseName like HelloWorld or let it be filled in implicitly, like TwoPlusTwo above.)

Once you fill in a working program, you need to decide on what parts you want to blank out for the student. Websheets supports two kinds of blanks:

For example, the Distance websheet has both kinds of blanks. Try typing some stuff in the blanks and see how they resize to accomodate unlimited student input.

Both kinds of blanks are indicated by using \[ to denote the start of the blank, and ]\ to denote the end of the blank. The interior region will be left blank and editable for the user.

The rules for how these delimiters must appear in the reference solution / template are:

For example, look at how Distance defines both kinds of blanks.

(In the multi-line case, additional whitespace on the lines containing the delimiters is currently ignored, although it could be meaningful in a later version, say where Python is used and we want to force indentation of a region.)

Hidden code, fake code, default contents for blanks

Feel free to skip this section if you're reading the documentation for the first time.

The simple \[...]\ delimiters are sufficient to describe a large variety of exercises. But to allow the exercise writer more flexibility, websheets allow a few additional kinds of regions.

You can use \hide[...]\ to enter code that will be hidden from the student's UI. It will be included in both the reference solution and the student's combined solution that gets executed upon submission. For examples, see

You can use \fake[...]\ to display read-only text in the user interface that looks like it is part of the program, but that won't actually be included. For examples, see

You can use \show: to have a blank initially populated with some text. (Think of providing a buggy program fragment for a student to fix.) To do this, change \[ blanked-out text ]\ to \[ blanked-out text \show: default text here ]\. For example, see

Cheatsheet: What elements appear where?

in UI?in reference solution?in executed/tested combined student solution?
plain text yes (read-only) yes yes
blanked-out \[...]\ yes (editable, initially blank) yes yes (edited student text spliced in)
\show: of blanked-out yes (editable) no yes (edited student text spliced in)
\fake[ ... ]\ yes (read-only)nono
\hide[ ... ]\ no yes yes

The exercise description (HTML)

You can use as few or as many HTML elements as you like in describing the exercise. The simplest descriptions are just text, using <p> to separate paragraphs, and sometimes using <tt> and <pre> to use a monospace font.

LaTeX. All websheets have MathJaX processing turned on. This means that you can include LaTeX using $inline format$ or $$big centered display format$$. Look at for examples of both.

Links. Sometimes you'll want to add a link to another websheet inside of the description. One way to do this nicely is to write a link with a URL like

<a href="javascript:websheets.load('java/00-intro/SquareSwap')">SquareSwap</a>
E.g. see .

Diagrams. You can use any software and any formats you like, but the example exercises using diagrams (e.g. ; ) were created using IPE (which uses PDFs as an editable format) and rasterized to PNG using render.sh.

Java test suite

For Java, the tests contains a snippet of Java code that makes calls to our testing API. These API calls cause the automatic grader to dynamically execute both the reference code and student code in separate namespaces, capturing their outputs and return values, and comparing the results for correctness.

Because tests will be spliced in to a Java program, any valid Java code can be used. For example, instead of writing

testMain("AA");
testMain("BB");
…
testMain("ZZ");
you could write
for (char ch='A'; ch<='Z'; ch++) testMain(""+ch+ch);
You could even define local or anonymous classes if you want (although no current examples do this).

Basics: testing static methods

The testing methods testMain() for testing main, test() for instance methods, testConstructor() for constructors, and testOn() for instance methods are used to execute student code and execute reference code and compare the results. It handles things like comparing return values, capturing and comparing printed output, catching and comparing exceptions, and looking for changes to arguments automatically.
testMain(String... args)
call main on these arguments.
E.g. see
test(String methodName, Object... args)
call a static method named methodName on this list of arguments
E.g. see
note: if there's exactly one argument, which is an array, Java requires you use
test(methodName, (Object) /* your array here */)
due to inherent ambiguity in varargs. E.g. see . The ambiguity doesn't exist if there are multiple arguments of which some are arrays, e.g. see .

If you're reading the manual for the first time, now would be a good time for you to try creating some sample exercises; you've now read the documentation for the essential mechanics.

Basics: testing instance methods

For testing instance methods, you'll generally call a constructor defined by the student, and then call an instance method. The typical approach involves 3 steps, like this sample from :
saveAs = "attendance"; // (1)
testConstructor(); // (2) i.e. attendance = new Clicker();
testOn("attendance", "curr"); // (3) i.e. attendance.curr(); 
This means,
saveAs = String  (default null)
save the value returned by the next test as a named object. (implementation note: we maintain two parallel namespaces for student and reference solutions.)
Note: you can save anything, not just constructor calls. E.g. in , we save a method's return value:
saveAs = "scaled";
testOn("x", "times", 10.0);
testConstructor(Object... args)
call the constructor with this list of arguments
testOn(String instanceName, String methodName, Object... args)
on the named instance, call the named instance method with this list of arguments
Here is one more tool used in conjunction with saveAs.
var(String)
once an object is saved, you may want to pass it an argument to a method, rather than just calling its instance methods. For example in , after constructing two objects x and y, we call x.plus(y) like so:
testOn("x", "plus", var("y"));
basically, the var() wrapper is needed to distinguish the saved variable y from the string literal "y"

Exceptions and standard input

expectException = boolean  (default false)
indicates that the student code should throw an exception on the following test. (it will catch whatever exception the reference code throws, and expect the student code to throw the same class of exception)
E.g., .
stdin = String  (default null)
for the next test, set the contents of standard input to the following String
E.g., .
stdinURL = String  (default null)
same as stdin, but fetch this webpage and use its contents. Useful when there is a long data set posted online, and you prefer not to copy a long text into the websheet definition
E.g., .
note: use simple string literals only. e.g. stdURL = "http://goo.gl/" + "robots.txt"; is bad, it will confuse the prefetcher that grabs the data before starting the sandbox.

Further methods

remark(String s)
Print out a remark to the user explaining what the tests are doing. Useful when you still want them to see the automatically-generated description of the test, but you also want to explain more context to the student. s should be valid HTML.
c.f. title = which replaces the automatically-generated test description.
E.g., .
E.g., .
store(Object o)
Copy an object to the user and reference namespace. Used for checking shallow copying in .
construct(String pckg, String clss, String type_prms, Object... args)
Use this to create a new instance of some other class. pckg should be a fully qualified package or null to use student/reference code; clss should be a simple class name; type_prms should be "" if you are not constructing a generic, or a list of type arguments for a generic if you are (this is just shown to the student in the description of the test since generic erasure happens at runtime); and args are the consturctor arguments. It is best seen by these two examples:
In we call
construct("stdlibpack", "Queue", "<Integer>")
to mimic a call to new Queue<Integer>() (the package stdlibpack has the COS 126 "standard" classes, and we want the Queue from that package)
In we call
construct(null, "Person", "", "Tony")
to mimic a call to new Person("Tony")Person is in the dependencies of that websheet. Leaving pckg as null means that the student's code uses the Person from the student package while the reference uses the one from the reference package.
testNamedMain(String methodToCall, String fakeClassName, String... args)
Call some hidden internal method, but pretend you are calling main(args) on some other testing class. Used in at testNamedMain("exampleClientMain", "ExampleClient"); to run the hidden method exampleClientMain but to tell the student we are running java ExampleClient.
Note: store(), construct() and testNamedMain() will reset all options to their defaults, like the basic test…() methods.

Further options

title = String  (default null)
replace the automatically-generated description of the next test with the given HTML. Useful if you are using a lot of fake or hidden code and the automatically-generated description the student would see in testing would make no sense.
c.f. remark() which adds extra commentary before the next test
E.g. .
E.g. .
quietOnPass = boolean  (default false)
Don't show the next test unless the student fails it. Especially useful if you have a lot of tests.
E.g. .
E.g. .
maxOutputBytes = int  (default 10000)
Change the maximum number of bytes the student code can print per test before it is terminated.
E.g. .
E.g. .
dontRunReference = boolean  (default false)
Don't run the reference solution. Instead, rely on either template code or hidden code that will determine if the student passes, and which will throw an exception if they failed, but not if they passed.
E.g. .
E.g. .
Note that hidden code can make thrown exceptions look a bit nicer by using websheets.Grader.FailException like FTPLimiter.
cloneForStudent = boolean  (default true)
Before passing arguments to the student, clone them. So if the student mutates the arguments, it doesn't affect the grader.
This is turned off to check shallow copying in .
cloneForReference = boolean  (default true)
Same as above, but for reference code. No current example.
realTolerance = double  (default 1E-4)
Relative error accepted for outputs. (Except if reference answer is 0, this is the absolute error accepted.) No current example.
ignoreRealFormatting = boolean  (default true)
If false, ignore realTolerance, also don't accept 5.00 as the same as 5.0. No current example.
ignoreTrailingSpaces = boolean  (default true)
Ignore trailing spaces at the end of each line. No current example.

Misc

defaultOptions
To set an option permanently (as opposed to just for one test), use syntax like
defaultOptions.quietOnPass = true;
E.g, see .
randgen
This is a Random instance just made available for convenience. E.g., see which calls randgen.nextInt(100) to generate a random number from 0 to 99.

Java-specific optional websheet properties

Here are a couple of other properties that are available for use in websheets
classname
when you want the visible and executed class name not to be the same as the name of the overall exercise (the Python module name, referred to internally as the slug).
E.g. .
dependencies
require student to complete some other websheet before this one; the student's most recent correct solution to the dependency will be made available when compiling executing this one.
E.g. .
Note: if instead you want to give student extra reference code without requiring them to actually complete the dependency, you can use \hide instead:
E.g. vs .
imports
a list of classes to import (will be made visible in UI and added to reference solution)
E.g. .

C++ test suite

Websheets supports two different C++ engines:

C++ (call main with stdin and args)

The more basic kind of C++ websheet is one where the student is developing a main function. We test their code by running the main function (running the program) one or more times.

For each run, you specify the standard input and the command-line arguments, both of which are optional. Specifically, for our editor, each test is a JSON dict with optional keys stdin (a string) and args (a list of strings), and the test suite is a list of tests.

Take a look at which uses both stdin and args. Many other C++ examples use just one of stdin or args but not both.

You can add "hidden":true to a test to make the input now be shown. You can add "title":"your title here" to label a test with a title. Thanks to dz0 for this feature!

A test that runs with no inputs is an empty dict, so a test suite that just runs one test with no inputs would have test suite [{}]. See .

C++ (call functions directly)

The other kind of C++ test engine supported is one that makes function calls directly. This allows more fine-grained testing of multiple functions. (Unlike Java, it doesn't allow stdin or calling OOP member functions.)

A test suite for this engine is a list of commands, where each command has one of two types:

  1. ["check-function", "function name", "return type", ["arg 1 type", "arg 2 type", …]]
    which checks that a function of this name and exact signature exists
  2. ["call-function", "function name", ["arg literal 1", "arg literal 2", …]]
    which calls the function and checks that its return value, when printed, is correct
See as well as .

C++-specific optional websheet properties

Is example?
Used to indicate a chunk of code that is being included (perhaps by embedding) just for demonstration purposes.
E.g. .
Examples can be editable, e.g. which is used to kick off an in-class discussion. Sometimes this is also useful when your exercise can't really be autograded properly due to technical constraints. See which uses randomness and which involves timeouts.

Multi-language optional websheet properties

Here are a couple of other properties that are available:
epilogue
show a message or explanation after problem is completed.
E.g. .
Forbidden substrings
Resrtict students from including these words in their code.
E.g. .