Skip to main content

Creating and Managing Rubrics

This guide explains how rubrics drive the hand-grading experience and how scores are computed. It also includes copy-pasteable YAML examples for each configuration option.
Rubrics can only be edited by instructors. When editing a rubric, you will be prompted to confirm before leaving the page if you have unsaved changes.

High-Level Overview

  • Rubric: Named collection of parts used for a specific review round (self-review, grading-review, meta-grading-review). The grading review is the main review that students will see.
  • Part: Logical section within a rubric (e.g., Code Quality, Functionality). Contains criteria. Parts can be configured for individual grading or assign-to-student modes for group submissions.
  • Criteria: A scoring rule block with:
    • is_additive and is_deduction_only: together these pick one of three scoring modes (see How Points Are Computed). is_deduction_only is required on every criteria by the shipped schema; is_deduction_only: true selects “Penalty only” whatever is_additive says, and is_additive is read only when is_deduction_only is false. The editor writes one mode at a time.
    • total_points: Max points for this criteria. It is not possible for a grader to assign more points to nested checks, even if you allow them to select that many points.
    • min_checks_per_submission / max_checks_per_submission: How many checks must/can be applied for the criteria.
    • One or more checks.
  • Check: The atom of feedback and scoring. A check can be global or an annotation applied to a file line or to an artifact. Checks can have:
    • points or selectable data.options with distinct labels/points
    • is_annotation + annotation_target (file or artifact), optional file/artifact to link
    • is_required: Must be applied by graders
    • is_comment_required: Comment is required when applying
    • max_annotations: Cap on times a check may be applied as annotations
    • student_visibility: always | if_applied | if_released | never
    • kpi_category: Optional link to repository analytics (commits, prs_opened, pr_review_comments, issues_opened, issues_closed, issue_comments)

Where This Shows Up in the Hand-Grader

  • The right-hand panel is the rubric sidebar. Global checks render as radio/checkbox options. Annotation checks are applied from the code view or artifact cards by right-clicking a line or selecting an artifact.
  • Each applied check creates a comment entry tied to a check with points and optional text.
  • For each criteria, the sidebar shows a running subtotal.

Artifact Rendering

When grading submissions, artifacts (non-code files) are displayed with appropriate rendering:
  • Markdown files (.md, .markdown): Rendered with full formatting including headers, lists, links, and code blocks
  • Plain text files (.txt): Displayed as formatted text for easy reading
  • Binary files: Available for download
This allows graders to provide feedback on documentation, reports, and other non-code deliverables directly within the grading interface. Annotation checks can target specific artifacts, enabling inline feedback on README files, design documents, and other text-based submissions.

How Points Are Computed

Each criteria uses one of three scoring modes. The rubric editor names them “Award per check”, “Deduct from total”, and “Penalty only”; in YAML they are two booleans. is_deduction_only: true wins outright — is_additive is consulted only when is_deduction_only is false. Both keys appear in the YAML column below because is_deduction_only is required on every criteria; copying a mode without it gives you a rubric the editor rejects.
“Penalty only” is the mode that is easy to miss. Unlike “Deduct from total” it does not begin from total_points, so a criteria with no checks applied contributes 0 rather than full marks, and its best possible contribution is 0. Here total_points acts as a floor on the penalty rather than a starting balance. Use this mode for penalties that should not inflate the rubric’s maximum, such as a late or academic-integrity deduction.
is_deduction_only is required on every criteria. The YAML source editor validates your rubric against the shipped RubricSchema.json and marks a criteria without the key as an error. While any error marker is present the editor stops re-parsing your YAML, so the live preview freezes until you add it. Set is_deduction_only: false for the two modes that are not “Penalty only”. is_additive is optional and defaults to false, which yields “Deduct from total”. If you want “Award per check” you must say is_additive: true explicitly.
  • The rubric total is the sum across criteria.
  • Scores automatically recalculate when rubric check scores change, ensuring submission grade totals stay synchronized with the latest rubric configuration.

Note on Options vs Base Points

  • If a check has data.options, the selected option’s points replaces the check’s base points when applied.
  • The base points still matters for the points summary. On an “Award per check” criteria, the summary counts each check’s base points, not its options, so set it to the highest option.
  • Options must be 2+ (single options are disallowed by the editor and schema).

References Across Rubrics

  • Checks can reference checks from other rubrics to surface related feedback when grading. This is managed inline in the sidebar (preview mode).

YAML Schema Quick Reference

These examples conform to the rubric schema the YAML editor validates against, and each one can be pasted in as it stands.

Minimal Rubric

Additive vs Subtractive Criteria

Check with Options (Multiple Choice)

Annotations (File vs Artifact) and Limits

Required Checks and Required Comments

Min/Max Checks per Criteria

Student Visibility

Drafting a Rubric with an AI Assistant

You can ask an AI assistant to draft a rubric from your assignment handout, then paste the YAML it produces into the YAML source editor. The assistant runs on your side. Pawtograder does not call a model, and the assistant does not need any student work to draft a rubric.

Why Coarse Buckets

The rubrics that work best here give each criteria one check with three or four options, and each option label describes what a grader sees in the submission. Two graders will agree on “every function has a purpose statement”. They will not agree on a 7 versus an 8 for “documentation quality”. Clear buckets make grading faster and more consistent, and they are the first step if you later want an AI assistant to suggest grades, since its suggestions can only be as consistent as the rubric it is given.

How to Draft One

1

Give the assistant the assignment

Paste in the handout, or point the assistant at the handout repository. If the assistant is connected to Pawtograder’s MCP server, get_handout_files and get_grader_files return the handout and solution repositories. See AI assistance for helping students for the setup. Anything the assistant reads is sent to its model provider, and get_grader_files sends your solution repository, so skip it if your course does not allow solutions to be shared with an outside service.
2

Ask for options in Pawtograder's YAML

For example:
3

Paste the YAML into the rubric editor

Open Configure Rubric, select the Grading Review tab, switch to YAML source, replace the contents with the assistant’s YAML, and click Save. The editor shows Rubric Saved when the rubric is stored.
4

Check the rubric before grading

Confirm that the Grading Rubric Points Summary reports the total you expect. The rubric and the autograder together should add up to the assignment’s max points, so if the autograder awards points, the rubric covers only the rest. Then grade three or four real submissions with the rubric. Wherever you hesitate between two options, rewrite their labels so the difference is something you can point to in the submission.

Example

This rubric is in the shape to ask for. It is worth 20 points and can be pasted in as it stands.
The example sets each check’s points to its highest option. Note on Options vs Base Points explains why.

Troubleshooting

  • The editor marks an error on a criteria. Every criteria needs is_deduction_only. See How Points Are Computed.
  • The editor rejects a check’s options. A check needs at least two options, or none.
  • Save fails with “Point values cannot be negative”. Use positive points and pick a scoring mode for deductions instead.
  • The points summary shows 0 for the rubric. The checks have points: 0. See Note on Options vs Base Points.

Editing Rubrics

When editing rubrics, Pawtograder protects against accidental data loss. If you attempt to leave the rubric editor with unsaved changes, a confirmation dialog appears warning you that your changes will be lost. You can choose to stay and save, or discard the changes and leave. This protection applies when:
  • Navigating away via links within the application
  • Using browser back/forward buttons
  • Closing the browser tab or window
  • Refreshing the page
  • Using the mobile navigation dropdown
The warning only appears when you have actual unsaved changes in the rubric editor. After successfully saving, the warning clears automatically.
If you’re working on rubrics across multiple tabs, only the currently visible tab will manage the unsaved changes warning to avoid false alerts.
After reviews are released to students, grading comments cannot be edited or deleted (except by instructors — see Grading Comment Restrictions After Release). Plan your feedback carefully before releasing grades.

How This Renders in the Hand-Grader

  • Global checks appear as radio buttons (when max_checks_per_submission = 1) or checkboxes (otherwise).
  • Checks with data.options render a choice list. The selected option’s label is prefixed to the comment.
  • Annotation checks are applied via right-click on a code line or by selecting an artifact.
  • Checks linked to specific artifacts display a clickable link to navigate directly to that artifact.
  • For each criteria, the sidebar shows:
    • Additive: earned / total_points
    • Subtractive: remaining / total_points

Regrade Requests

Students can request regrades on rubric items where they believe the grading was incorrect. Regrade requests are automatically prevented on self-review rubric items, since those are completed by the students themselves.

Split-Rubric Group Grading

For group assignments, you can apply per-student adjustments when using split-rubric grading. This allows you to assign different scores or provide individualized feedback to group members while maintaining the efficiency of group grading workflows.

Editing Rubric Scores After Completion

TAs and instructors can edit or delete rubric scores even after completing a review. This allows for:
  • Correcting mistakes in grading
  • Adjusting scores based on regrade requests
  • Updating feedback after discussion with students
  • Modifying annotations or comments

How to Edit After Completion

  1. Navigate to the completed review
  2. Click on any rubric score or annotation
  3. Make your changes (modify points, update comments, or delete checks)
  4. Changes are saved automatically
  5. The review remains marked as complete unless you explicitly mark it incomplete
Editing completed reviews does not automatically notify students. If grades have been released, you may need to communicate changes to affected students.

Notes and Tips

  • Keep criteria focused and keep check names short; longer explanation should go in description.
  • Use max_annotations to prevent over-counting nitpicks.
  • Prefer options when the same check has graded tiers (e.g., Complete/Partial/Incomplete).
  • Use student_visibility to separate internal notes from student-facing feedback.
  • Grade override notes are now properly displayed in the grading interface.
  • Rubric scores are automatically recalculated when check scores change, ensuring the gradebook stays up-to-date.
  • Save frequently to avoid losing work, especially for complex rubrics.

Field Reference: Required vs Optional (with defaults)

Rubric
  • Mandatory: name, parts
  • Optional: description
Part
  • Mandatory: name, criteria
  • Optional: id, description, data
Criteria
  • Mandatory: name, checks, is_deduction_only (schema-required; use false unless the criteria is “Penalty only”)
  • Optional: id, description, data, is_additive (default: false), total_points (default: 0), min_checks_per_submission, max_checks_per_submission
Check
  • Mandatory: name, is_annotation (boolean), is_required (boolean), is_comment_required (boolean), points (number; on an “Award per check” criteria with data.options, set it to the highest option’s points, or the points summary counts the check as 0)
  • Optional: id, description, file, artifact, annotation_target (file | artifact; default behavior in UI is file when omitted for annotations), max_annotations, data (see below), student_visibility (default: always)
Check data.options (for multiple-choice checks)
  • Mandatory per option: label, points
  • Optional per option: description
  • Notes: Must define at least two options (single-option checks are rejected by the editor). When options are present, the selected option’s points replace the base points for that check when applied.
Defaults and Behaviors Used by the Grader UI
  • Criteria without is_additive are treated as subtractive (deduction) criteria.
  • Criteria without total_points default to 0 (i.e., no contribution unless checks add points in additive mode).
  • Check student_visibility defaults to always.
  • Annotation annotation_target defaults to file in the UI if omitted.
  • min_checks_per_submission and max_checks_per_submission are optional; when not set, graders are not constrained by count.

Grading Comment Restrictions After Release

Once a review has been released to students, grading comments are restricted to maintain grading integrity:
  • Graders can only edit or delete their own comments after a review is released
  • Instructors retain full editing privileges and can modify any comment, even after release
  • This prevents graders from modifying feedback after students have seen it, while allowing instructors to make corrections if needed
This restriction applies to all rubric comments and annotations, ensuring that students see consistent feedback once grades are released.

Check References (Cross-Rubric Context)

What It Is

A check on one rubric can reference a check on another rubric. During grading, any applied feedback (comments/points) from the referenced check(s) is shown inline under the current check as “Related Feedback from Other Reviews”. This gives graders context from, e.g., self-review or prior review rounds.

Scoring and Visibility

  • References are informational only. They do not contribute points to the current rubric’s criteria and do not alter the score computation.
  • Referenced feedback is surfaced to graders in grading mode. Student visibility continues to follow each original check’s own student_visibility and release state within its source review.

How to Configure

  1. Navigate to the rubric preview/editor for an assignment.
  2. For the check you want to augment, click “Add Reference”.
  3. Search/select a check from other rubrics (the current rubric’s checks are excluded).
  4. Save. The relationship is stored so that, when grading, the referenced feedback appears under the referencing check.

Good Use Cases

  • Show a student’s self-review evidence next to the corresponding grading check.
  • Pull in meta-grading notes when doing final pass reviews.

Individual and Assign-to-Student Grading Modes

For group submissions, rubric parts can be configured with special grading modes to assess individual contributions:

Individual Grading Mode

When a rubric part has is_individual_grading: true, that part is graded separately for each group member. The grading interface displays the part once per student, and graders apply checks individually for each team member. Use cases:
  • Individual code quality assessments in group projects
  • Personal reflection questions
  • Individual contribution evaluations
Configuration:
When grading, you’ll see this part repeated for each group member with a student name badge. Annotations and checks must be applied separately for each student.

Assign-to-Student Mode

When a rubric part has is_assign_to_student: true, the grader selects which student the part applies to (or skips it if not applicable). Use cases:
  • Role-specific responsibilities (e.g., “Team Lead” duties)
  • Optional bonus sections that only some students completed
  • Targeted feedback for specific contributions
Configuration:
When grading, you’ll see a dropdown to select which student this part applies to, or choose “Skip” if it doesn’t apply to anyone.

Scoring with Individual/Assign-to-Student Parts

When a rubric includes individual or assign-to-student parts:
  • Shared score: Points from regular (whole-group) rubric parts, autograder, and tweak
  • Individual score: Points from individual-grading and assign-to-student parts for each student
  • Total per student: Shared + individual, capped to assignment maximum if configured
The grading interface displays a “Scores by student” breakdown showing shared, individual, and total scores for each group member.

Important Notes

  • A part cannot be both is_individual_grading and is_assign_to_student (mutually exclusive)
  • Individual grading parts require every rubric-linked comment to name the student it applies to
  • Assign-to-student parts are skipped in completion validation when not assigned to any student
  • Students only see their own individual scores, not their teammates’ scores