Introduction

IMPORTANT! I just deleted this entire book. The framework is changing quite a bit and it needs to be rewriten. More coming soon.

Rubrics

Rubrics are the main component of this framework. A rubric is simply a list of criteria. They are the first thing you should write down when writing a grader.

Within this framework, rubrics are represented with .yml files in the rubrics/ directory. "YAML" is a markup language that you can read about here. It's pretty common within sysadmin tools and chances are you're already familiar with it. We can use YAML to write out the details of our criteria.

A Practical Example

Rubrics are very easy to understand if you see one.

In the git_lab example, we wrote a hypothetical assignment for our hypothetical students. This lab is meant to teach our students about git and how to use it. We came up with 4 criteria that will make up our assignment. These criteria are

1. The student must install git and have it available at the command line
2. The student must initialize git in a directory
3. The student must make at least 3 commits in that repository
4. The student must push the repository to Github and have it publicly accessible

This is a great start. All we need to do is formalize this into a rubric.

# Our assignments name
name: Git Lab
# An optional description
desc: This lab is meant to teach the basics of Git and Github


# A list of our criteria
criteria:
  "Git installed":
    stub: git-installed
    desc: Git should be installed and accessible
    worth: 25
    messages: ["installed", "not installed"]

  "Git initialized in repo":
    stub: git-init
    desc: Current directory should have git initialized
    worth: 25
    messages: ["initialized", "not initialized"]

  "Commits present":
    stub: commits
    worth: 25
    desc: Current git repository should have more than 2 commits

  "Repo pushed":
    stub: pushed
    worth: 25
    desc: Current git repository should be pushed to github

You can see that each criteria has a name, a point value, and some other configuration options. The only required fields are the name and worth, everything else has defaults.

Lets dissect one of the criteria

# ...
criteria:
  # The criteria name
  "Git installed":
    # a stub is an identifier, it must be unique.
    # if you omit this, the stub will be the criterion's name,
    # lower-cased and whitespace replaced with dashes
    stub: git-installed
    # Just a description. Students will see this when grading
    desc: Git should be installed and accessible
    # The point value, can be any number, even negative.
    worth: 25
    # Success/failure messages. These default to "passed" and "failed".
    # The're just some extra information for the student
    messages: ["installed", "not installed"]

There are a few more options that you can provide, which you can read about on the Rubric Specification Page.

Loading a Rubric

After writing out a rubric in a .yml file, we can load it into our grader and use it.

The first step to loading a rubric is reading the file. This is done with the yaml! macro.

let rubric_yaml = yaml!("../rubrics/main.yml").expect("Couldn't load file!");

yaml! loads a file relative to the current file. It returns a Result, so we can deal with the error if we want. It's usually a good idea to call expect() so the program crashes if the file couldn't be read.

yaml! is special because it embeds the .yml in the compiled executable. This means you don't have to distribute the rubric file with the executable. Your rubric can also be kept private if you want it to be.

Next, we just have to pass the YAML data to the Rubric struct like this

let rubric = Rubric::from_yaml(&rubric_yaml).expect("Bad YAML!");

Note: We're using expect() again here. We want the program to crash at compile-time when we're working on the grader, not at run-time when the students are using it. Better for us to deal with the error than them.

Writing The Tests

We have a rubric loaded, but it has no way to actually verify that the criteria have been fulfilled. We're going to write one function for each of the criteria. The function (called a "criteria test" or just "test") has the responsibility of ensuring the criteria was actually fulfilled by the student.

Assignments obviously vary greatly in scope and material, so writing these tests is the bulk of the work to be done when writing a grader. It all depends on what the student should be doing.

Things like installing Git or running a web server are easy to verify, but other tasks might not be. Because every criteria has a function, the full force of Rust is on your side. You may have to get creative in writing your tests. You can always look at the examples on Github for inspiration.

See the Criteria Tests page for more information on this topic.

Rubric Specification

If you're looking for the syntax of the YAML language itself, then look here. This page is for the allowed items in a rubric's .yml file.

Minimal Rubric

Here is a rubric with as few items as possible. Everything here is required.

name: My rubric

criteria:
  "first and only criterion":
    worth: 10

All Items

Here is a full rubric with everything specified, with comments for more information about each key.

# Required name
name: My rubric
# Optional description. Gets shown to the student when grading
desc: Description of my rubric
# Sanity check. If the sum of all criteria doesn't add to this number,
# an error message will be displayed. Just ensures that you give the correct
# worth to all criteria
total: 100

criteria:
  "First criterion":
    # Can be any string, as long as it's unique.
    # If not specified, the stub will be the criterion's name,
    # lowercased and whitespace replaced with dashes
    stub: whatever-stub
    # Any number (even negative). Lowest number is run first.
    # Criteria without indices do not have consistent order
    index: 1
    # A description
    desc: You should do this to fulfil this criterion
    # required point value
    # can be negative
    worth: 50
    # success and failure messages
    # default to "passed" and "failed"
    messages: ["Passed!", "not passed"]
    # This will prevent the criterion from being displayed
    # to the student. Useful if you want hidden requirements 
    # or are grading a test
    # true or false
    hide: false

  # This criterion has all default values
  "Second criterion":
    stub: second-criterion
    index: 100
    desc: ""
    worth: 0
    messages: ["passed", "failed"]
    hide: false

Criteria Tests

As stated on the Rubrics page, a criteria tests is a Rust function that verifies if a criteria was actually fulfilled. They will vary widely. This page will show how to write a test and a few examples to get you started.

A Single Test

Tests are just functions, but they must have a specific signature. They must always accept the same parameters and return the same type of value. Here's the signature

// Be sure TestData is imported
use rubric::TestData;

fn my_test(data: &TestData) -> bool {
    // Test code goes here...
}

Every test must accept a reference to a TestData struct. This TestData is stored on a Submission, which I'll cover in a different section. What you should know now is that it's an alias to HashMap<String, String>.

Sometimes you won't need TestData in a test, in which case you can just name the parameter _ and Rust won't complain.

Tests must also return a boolean. true if it passes, false otherwise. If a test returns true, then the associated criteria's worth will be added to the point total. If all the criteria tests return true, the maximum score is achieved.

Using TestData

Remember that a TestData struct is really just a HashMap. It will contains keys and values that you specify when setting up a Submission. You can use any of the methods that HashMap's have. 90% of the time, you'll just want to read a value from the TestData. There's 2 ways to do that.

fn some_test(data: &TestData) -> bool {
    // The easy but dangerous way to get a value
    // this will *crash* if the key doesn't exist
    let my_value = data["my_key"];

    // The safe way to get a value
    if let Some(value) = data.get("my_key") {
        // Key exists, now we have the value
        println!("Value is {}", value);
    } else {
        // Key doesn't exist, something went wrong, handle error
        println!("Value doesn't exist!");
    }

    // ...
}

It's important that you take precautions when writing a grader. You really don't want it to crash while your students are running it. The two examples above to the same thing, but the second method won't crash if the key doesn't exist.

Organization

I strongly recommend making a test.rs file alongside main.rs to keep your tests in. Of course, you don't have to. You could keep your tests as loose functions in main.rs, or maybe have a submodule in main.rs.

Again, I recommend making a tests.rs file and keeping them in there. Here's how I set things up.

// tests.rs
use rubric::TestData;

fn test_from_tests_rs(_: &TestData) -> bool {
    // test code goes here
    return true;
}

// more tests here...

// main.rs
extern crate rubric;

// declare tests.rs as a module
mod tests;
// bring all test functions into scope
use tests::*;
// ...

Attaching Tests

Each criteria has an associated test, but we need to tell our program which test goes with which criteria. After we've written our tests and loaded our rubric, we can use the attach! macro to assign them. Just point the criteria's stub that you specified in the .yml file to the function name.

fn my_criteria_test(_: &TestData) -> bool {
    // test code goes here...
    return true;
}


fn main() {
    // load rubric
    // code omitted
    // be sure it's mutable
    let mut rubric = //...

    attach! {
        rubric,
        "criteria-stub" => my_criteria_test
    };
}

Helpers

There are a few helper modules and functions that perform some common tasks. Sometimes your tests will be one-liners from the helper modules. See the helpers module documentation on docs.rs for more info.

Examples

Some basic examples can be found in the examples directory on Github, specifically in this file in the git_lab example.

Submission

A submission represents one students work on an assignment. It contains any data you may want about the student, like name and ID, their grade, which criteria they passed and failed, information about their system, information they provide, etc.

A submission is the other half to a rubric. While rubrics are identical among all students, every student has their own unique submission. The stucture of a submission is all the same, but the values contained in it are unique to each student.

A submission is designed to be constructed during the grading process and sent back to you, the instructor. You'll end up with a list of submissions from every student containing their grade and all the data you specified.

Let's take a gander at a blank Submission struct to see what's in it:

Submission {
    time: 2020-08-04 Tue 21:13:34 -05:00,
    grade: 0,
    data: {},
    passed: [],
    failed: [],
}

There's a few default values like a timestamp, grade, and which criteria the submission passed and failed, but it's pretty empty. The important bit is the data HashMap. This is where all of the data you specify will live. All of this data will be collected when the student runs the grader, and will be sent back as part of the grade report.

The data! macro

I mentioned that the data field on a submission is a HashMap. This is sort of true. It's technically a TestData, which is an alias to HashMap<String, String>.

If you read the Criteria Tests section, you might remember TestData as the value that criteria tests must accept as a parameter. The TestData on a submission is what will be passed into these tests.

Rust does not have object literals the same way a language like Python does. Instead, this crate provides a data! macro that will create a TestData for you. Here's how to use it.

let data = data! {
    "some_key" => "some value",
    "other_key" => "other value"
};

Since TestData is an alias to HashMap<String, String>, the keys and values must be strings.

Creating a Submission

Creating a submission is easy

// creates a blank submission like the one above
let sub = Submission::new();

You will most likely want to create a submission with some data. You can do that like this.

let data = data! {
    "key" => "value"
};

let sub = Submission::from_data(data);

// This will create the following
Submission {
    time: 2020-08-04 Tue 21:13:34 -05:00,
    grade: 0,
    data: {
        "key": "value",
    },
    passed: [],
    failed: [],
}

Prompting for Data

When creating a submission, you'll almost always want to ask the student for some information. Usually their name and ID, but maybe some other information as well.

The prompt! macro does this easily.

fn main() {
    // Asks for something, enforces the correct type
    let name = prompt!("Name: ", String);
    // Will crash if they enter anything but a number
    let age = prompt!("Age: ", usize);
    // Will crash if they don't enter a valid IPv4 address
    let ip = prompt!("IP Addr: ", std::net::IPv4Addr);
}

You can combine this with the data! macro to easily collect information from the user and encapsulate it in a submission.

fn main() {
    let data = data! {
        "name" => prompt!("Name: ", String),
        "id"   => prompt!("ID: ", String)
    }
}

Note: because TestData must contain string values, you lose out on the type enforcement that prompt! provides. This is an unfortunate side effect of the TestData type; all values must be strings.

There is also a prompt function under helpers::cli::prompt which will loop until it gets a valid value, but it does not have the same type enforcement feature that prompt! has; it only returns strings.