Introduction

Setup

This is a Rust crate and can be installed like any other Rust crate. You'll need Rust and it's build tool, cargo installed. Read more about that here.

You can create a new Rust project with

cargo new --bin my_grader

In the Cargo.toml file that cargo creates for you, add this crate as a dependency.

# ...
[dependencies]
rubric = "0.12" # or latest version

In your main.rs, delcare the crate with the #[macro_use] configuration flag. You can import just the items you need (recommended), or import everything with *.

#[macro_use] extern crate rubric;

// import what you need
use rubric::{Rubric, Submission, TestData, /* ... */};
// or everything
use rubric::*;

I would also recommend creating a rubrics/ directory alongside your src/ directory to store your rubrics.

Rubrics

Rubrics are the main component of this framework. A rubric is simply a list of criteria. They are the first thing you should write down when writing a grader.

Within this framework, rubrics are represented with .yml files in the rubrics/ directory. "YAML" is a markup language that you can read about here. It's pretty common within sysadmin tools and chances are you're already familiar with it. We can use YAML to write out the details of our criteria.

A Practical Example

Rubrics are very easy to understand if you see one.

In the git_lab example, we wrote a hypothetical assignment for our hypothetical students. This lab is meant to teach our students about git and how to use it. We came up with 4 criteria that will make up our assignment. These criteria are

1. The student must install git and have it available at the command line
2. The student must initialize git in a directory
3. The student must make at least 3 commits in that repository
4. The student must push the repository to Github and have it publicly accessible

This is a great start. All we need to do is formalize this into a rubric.

# Our assignments name
name: Git Lab
# An optional description
desc: This lab is meant to teach the basics of Git and Github


# A list of our criteria
criteria:
  "Git installed":
    stub: git-installed
    desc: Git should be installed and accessible
    worth: 25
    messages: ["installed", "not installed"]

  "Git initialized in repo":
    stub: git-init
    desc: Current directory should have git initialized
    worth: 25
    messages: ["initialized", "not initialized"]

  "Commits present":
    stub: commits
    worth: 25
    desc: Current git repository should have more than 2 commits

  "Repo pushed":
    stub: pushed
    worth: 25
    desc: Current git repository should be pushed to github

You can see that each criteria has a name, a point value, and some other configuration options. The only required fields are the name and worth, everything else has defaults.

Lets dissect one of the criteria

# ...
criteria:
  # The criteria name
  "Git installed":
    # a stub is an identifier, it must be unique.
    # if you omit this, the stub will be the criterion's name,
    # lower-cased and whitespace replaced with dashes
    stub: git-installed
    # Just a description. Students will see this when grading
    desc: Git should be installed and accessible
    # The point value, can be any number, even negative.
    worth: 25
    # Success/failure messages. These default to "passed" and "failed".
    # The're just some extra information for the student
    messages: ["installed", "not installed"]

There are a few more options that you can provide, which you can read about on the Rubric Specification Page.

Loading a Rubric

After writing out a rubric in a .yml file, we can load it into our grader and use it.

The first step to loading a rubric is reading the file. This is done with the yaml! macro.

let rubric_yaml = yaml!("../rubrics/main.yml").expect("Couldn't load file!");

yaml! loads a file relative to the current file. It returns a Result, so we can deal with the error if we want. It's usually a good idea to call expect() so the program crashes if the file couldn't be read.

yaml! is special because it embeds the .yml in the compiled executable. This means you don't have to distribute the rubric file with the executable. Your rubric can also be kept private if you want it to be.

Next, we just have to pass the YAML data to the Rubric struct like this

let rubric = Rubric::from_yaml(&rubric_yaml).expect("Bad YAML!");

Note: We're using expect() again here. We want the program to crash at compile-time when we're working on the grader, not at run-time when the students are using it. Better for us to deal with the error than them.

Writing The Tests

We have a rubric loaded, but it has no way to actually verify that the criteria have been fulfilled. We're going to write one function for each of the criteria. The function (called a "criteria test" or just "test") has the responsibility of ensuring the criteria was actually fulfilled by the student.

Assignments obviously vary greatly in scope and material, so writing these tests is the bulk of the work to be done when writing a grader. It all depends on what the student should be doing.

Things like installing Git or running a web server are easy to verify, but other tasks might not be. Because every criteria has a function, the full force of Rust is on your side. You may have to get creative in writing your tests. You can always look at the examples on Github for inspiration.

See the Criteria Tests page for more information on this topic.

Rubric Specification

If you're looking for the syntax of the YAML language itself, then look here. This page is for the allowed items in a rubric's .yml file.

Minimal Rubric

Here is a rubric with as few items as possible. Everything here is required.

name: My rubric

criteria:
  "first and only criterion":
    worth: 10

All Items

Here is a full rubric with everything specified, with comments for more information about each key.

# Required name
name: My rubric
# Optional description. Gets shown to the student when grading
desc: Description of my rubric
# Sanity check. If the sum of all criteria doesn't add to this number,
# an error message will be displayed. Just ensures that you give the correct
# worth to all criteria
total: 100

criteria:
  "First criterion":
    # Can be any string, as long as it's unique.
    # If not specified, the stub will be the criterion's name,
    # lowercased and whitespace replaced with dashes
    stub: whatever-stub
    # Any number (even negative). Lowest number is run first.
    # Criteria without indices do not have consistent order
    index: 1
    # A description
    desc: You should do this to fulfil this criterion
    # required point value
    # can be negative
    worth: 50
    # success and failure messages
    # default to "passed" and "failed"
    messages: ["Passed!", "not passed"]
    # This will prevent the criterion from being displayed
    # to the student. Useful if you want hidden requirements 
    # or are grading a test
    # true or false
    hide: false

  # This criterion has all default values
  "Second criterion":
    stub: second-criterion
    index: 100
    desc: ""
    worth: 0
    messages: ["passed", "failed"]
    hide: false

Criteria Tests

As stated on the Rubrics page, a criteria tests is a Rust function that verifies if a criteria was actually fulfilled. They will vary widely. This page will show how to write a test and a few examples to get you started.

A Single Test

Tests are just functions, but they must have a specific signature. They must always accept the same parameters and return the same type of value. Here's the signature

// Be sure TestData is imported
use rubric::TestData;

fn my_test(data: &TestData) -> bool {
    // Test code goes here...
}

Every test must accept a reference to a TestData struct. This TestData is stored on a Submission, which I'll cover in a different section. What you should know now is that it's an alias to HashMap<String, String>.

Sometimes you won't need TestData in a test, in which case you can just name the parameter _ and Rust won't complain.

Tests must also return a boolean. true if it passes, false otherwise. If a test returns true, then the associated criteria's worth will be added to the point total. If all the criteria tests return true, the maximum score is achieved.

Using TestData

Remember that a TestData struct is really just a HashMap. It will contains keys and values that you specify when setting up a Submission. You can use any of the methods that HashMap's have. 90% of the time, you'll just want to read a value from the TestData. There's 2 ways to do that.

fn some_test(data: &TestData) -> bool {
    // The easy but dangerous way to get a value
    // this will *crash* if the key doesn't exist
    let my_value = data["my_key"];

    // The safe way to get a value
    if let Some(value) = data.get("my_key") {
        // Key exists, now we have the value
        println!("Value is {}", value);
    } else {
        // Key doesn't exist, something went wrong, handle error
        println!("Value doesn't exist!");
    }

    // ...
}

It's important that you take precautions when writing a grader. You really don't want it to crash while your students are running it. The two examples above to the same thing, but the second method won't crash if the key doesn't exist.

Organization

I strongly recommend making a test.rs file alongside main.rs to keep your tests in. Of course, you don't have to. You could keep your tests as loose functions in main.rs, or maybe have a submodule in main.rs.

Again, I recommend making a tests.rs file and keeping them in there. Here's how I set things up.

// tests.rs
use rubric::TestData;

fn test_from_tests_rs(_: &TestData) -> bool {
    // test code goes here
    return true;
}

// more tests here...

// main.rs
extern crate rubric;

// declare tests.rs as a module
mod tests;
// bring all test functions into scope
use tests::*;
// ...

Attaching Tests

Each criteria has an associated test, but we need to tell our program which test goes with which criteria. After we've written our tests and loaded our rubric, we can use the attach! macro to assign them. Just point the criteria's stub that you specified in the .yml file to the function name.

fn my_criteria_test(_: &TestData) -> bool {
    // test code goes here...
    return true;
}


fn main() {
    // load rubric
    // code omitted
    // be sure it's mutable
    let mut rubric = //...

    attach! {
        rubric,
        "criteria-stub" => my_criteria_test
    };
}

Helpers

There are a few helper modules and functions that perform some common tasks. Sometimes your tests will be one-liners from the helper modules. See the helpers module documentation on docs.rs for more info.

Examples

Some basic examples can be found in the examples directory on Github, specifically in this file in the git_lab example.

Submission

A submission represents one students work on an assignment. It contains any data you may want about the student, like name and ID, their grade, which criteria they passed and failed, information about their system, information they provide, etc.

A submission is the other half to a rubric. While rubrics are identical among all students, every student has their own unique submission. The stucture of a submission is all the same, but the values contained in it are unique to each student.

A submission is designed to be constructed during the grading process and sent back to you, the instructor. You'll end up with a list of submissions from every student containing their grade and all the data you specified.

Let's take a gander at a blank Submission struct to see what's in it:

Submission {
    time: 2020-08-04 Tue 21:13:34 -05:00,
    grade: 0,
    data: {},
    passed: [],
    failed: [],
}

There's a few default values like a timestamp, grade, and which criteria the submission passed and failed, but it's pretty empty. The important bit is the data HashMap. This is where all of the data you specify will live. All of this data will be collected when the student runs the grader, and will be sent back as part of the grade report.

The data! macro

I mentioned that the data field on a submission is a HashMap. This is sort of true. It's technically a TestData, which is an alias to HashMap<String, String>.

If you read the Criteria Tests section, you might remember TestData as the value that criteria tests must accept as a parameter. The TestData on a submission is what will be passed into these tests.

Rust does not have object literals the same way a language like Python does. Instead, this crate provides a data! macro that will create a TestData for you. Here's how to use it.

let data = data! {
    "some_key" => "some value",
    "other_key" => "other value"
};

Since TestData is an alias to HashMap<String, String>, the keys and values must be strings.

Creating a Submission

Creating a submission is easy

// creates a blank submission like the one above
let sub = Submission::new();

You will most likely want to create a submission with some data. You can do that like this.

let data = data! {
    "key" => "value"
};

let sub = Submission::from_data(data);

// This will create the following
Submission {
    time: 2020-08-04 Tue 21:13:34 -05:00,
    grade: 0,
    data: {
        "key": "value",
    },
    passed: [],
    failed: [],
}

Prompting for Data

When creating a submission, you'll almost always want to ask the student for some information. Usually their name and ID, but maybe some other information as well.

The prompt! macro does this easily.

fn main() {
    // Asks for something, enforces the correct type
    let name = prompt!("Name: ", String);
    // Will loop until they enter a number
    let age = prompt!("Age: ", usize);
    // Will loop until they enter a valid IPv4 address
    let ip = prompt!("IP Addr: ", std::net::Ipv4Addr);
}

You can combine this with the data! macro to easily collect information from the user and encapsulate it in a submission.

fn main() {
    let data = data! {
        "name" => prompt!("Name: ", String),
        "id"   => prompt!("ID: ", String)
    }
}

Note: because TestData must contain string values, you lose out on the type enforcement that prompt! provides. This is an unfortunate side effect of the TestData type; all values must be strings.

Dropbox

A dropbox is a place to send submissions after they're done being graded. It recieves submissions in JSON format and writes them to a CSV file for review by the instructor.

The dropbox is a web server run by you, the instructor. It should be run on a publicly available server with a static IP or DNS name. The web server comes preconfigured, all you need to do is give it an environment to run it.

You can open the dropbox by calling the dropbox::open(PORT) method, where PORT is a valid port number.

extern crate rubric;
use rubric::dropbox;

fn main() {
    // Runs the dropbox on port 8080
    dropbox::open(8080);
}

You'll see the following output

Dropbox is open! accepting POST requests to /submit
�🔧 Configured for development.
    => address: 0.0.0.0
    => port: 8080
    => log: normal
    => workers: 24
    => secret key: generated
    => limits: forms = 32KiB
    => keep-alive: 5s
    => tls: disabled
🛰  Mounting /:
    => GET / (return_ok)
    => POST /submit application/json (accept_submission)
��� Rocket has launched from http://0.0.0.0:8080       

The home route (/) should return an OK status, but no content. If you visit the url of your webserver, you should get a blank web page. This is good, it means everything is working properly.

Submitting to the dropbox

The post_json() method in the helpers::web module is made with the dropbox in mind. After creating and grading a Submission, just pass it and the url of your dropbox to send the submission.

extern crate rubric;
use rubric::{Submission, dropbox, helpers::web};

fn main() {
    let submission = Submission::new();
    
    // grade...

    // assuming your dropbox is running at this url
    let url = "http://my.dns.name.or.ip.com:8080/submit";

    // Submit and give some feedback
    match web::post_json(&url, &sub) {
        Ok(_)  => println!("Submission recorded!"),
        Err(e) => println!("Something went wrong! {}", e),
    };
}