Creating and Managing Rubrics
This guide explains how rubrics drive the hand-grading experience and how scores are computed. It also includes copy-pasteable YAML examples for each configuration option.Rubrics can only be edited by instructors. When editing a rubric, you will be prompted to confirm before leaving the page if you have unsaved changes.
High-Level Overview
- Rubric: Named collection of parts used for a specific review round (self-review, grading-review, meta-grading-review). The grading review is the main review that students will see.
- Part: Logical section within a rubric (e.g., Code Quality, Functionality). Contains criteria. Parts can be configured for individual grading or assign-to-student modes for group submissions.
- Criteria: A scoring rule block with:
is_additiveandis_deduction_only: together these pick one of three scoring modes (see How Points Are Computed).is_deduction_onlyis required on every criteria by the shipped schema;is_deduction_only: trueselects “Penalty only” whateveris_additivesays, andis_additiveis read only whenis_deduction_onlyisfalse. The editor writes one mode at a time.total_points: Max points for this criteria. It is not possible for a grader to assign more points to nested checks, even if you allow them to select that many points.min_checks_per_submission/max_checks_per_submission: How many checks must/can be applied for the criteria.- One or more checks.
- Check: The atom of feedback and scoring. A check can be global or an annotation applied to a file line or to an artifact. Checks can have:
pointsor selectabledata.optionswith distinct labels/pointsis_annotation+annotation_target(fileorartifact), optionalfile/artifactto linkis_required: Must be applied by gradersis_comment_required: Comment is required when applyingmax_annotations: Cap on times a check may be applied as annotationsstudent_visibility:always|if_applied|if_released|neverkpi_category: Optional link to repository analytics (commits, prs_opened, pr_review_comments, issues_opened, issues_closed, issue_comments)
Where This Shows Up in the Hand-Grader
- The right-hand panel is the rubric sidebar. Global checks render as radio/checkbox options. Annotation checks are applied from the code view or artifact cards by right-clicking a line or selecting an artifact.
- Each applied check creates a comment entry tied to a check with points and optional text.
- For each criteria, the sidebar shows a running subtotal.
Artifact Rendering
When grading submissions, artifacts (non-code files) are displayed with appropriate rendering:- Markdown files (
.md,.markdown): Rendered with full formatting including headers, lists, links, and code blocks - Plain text files (
.txt): Displayed as formatted text for easy reading - Binary files: Available for download
How Points Are Computed
Each criteria uses one of three scoring modes. The rubric editor names them “Award per check”, “Deduct from total”, and “Penalty only”; in YAML they are two booleans.is_deduction_only: true wins outright — is_additive is consulted only when is_deduction_only is false. Both keys appear in the YAML column below because is_deduction_only is required on every criteria; copying a mode without it gives you a rubric the editor rejects.
“Penalty only” is the mode that is easy to miss. Unlike “Deduct from total” it does not begin from
total_points, so a criteria with no checks applied contributes 0 rather than full marks, and its best possible contribution is 0. Here total_points acts as a floor on the penalty rather than a starting balance. Use this mode for penalties that should not inflate the rubric’s maximum, such as a late or academic-integrity deduction.is_deduction_only is required on every criteria. The YAML source editor validates your rubric against the shipped RubricSchema.json and marks a criteria without the key as an error. While any error marker is present the editor stops re-parsing your YAML, so the live preview freezes until you add it. Set is_deduction_only: false for the two modes that are not “Penalty only”.
is_additive is optional and defaults to false, which yields “Deduct from total”. If you want “Award per check” you must say is_additive: true explicitly.
- The rubric total is the sum across criteria.
- Scores automatically recalculate when rubric check scores change, ensuring submission grade totals stay synchronized with the latest rubric configuration.
Note on Options vs Base Points
- If a check has
data.options, the selected option’spointsreplaces the check’s basepointswhen applied. - The base
pointsstill matters for the points summary. On an “Award per check” criteria, the summary counts each check’s basepoints, not its options, so set it to the highest option. - Options must be 2+ (single options are disallowed by the editor and schema).
References Across Rubrics
- Checks can reference checks from other rubrics to surface related feedback when grading. This is managed inline in the sidebar (preview mode).
YAML Schema Quick Reference
These examples conform to the rubric schema the YAML editor validates against, and each one can be pasted in as it stands.Minimal Rubric
Additive vs Subtractive Criteria
Check with Options (Multiple Choice)
Annotations (File vs Artifact) and Limits
Required Checks and Required Comments
Min/Max Checks per Criteria
Student Visibility
Drafting a Rubric with an AI Assistant
You can ask an AI assistant to draft a rubric from your assignment handout, then paste the YAML it produces into the YAML source editor. The assistant runs on your side. Pawtograder does not call a model, and the assistant does not need any student work to draft a rubric.Why Coarse Buckets
The rubrics that work best here give each criteria one check with three or four options, and each option label describes what a grader sees in the submission. Two graders will agree on “every function has a purpose statement”. They will not agree on a 7 versus an 8 for “documentation quality”. Clear buckets make grading faster and more consistent, and they are the first step if you later want an AI assistant to suggest grades, since its suggestions can only be as consistent as the rubric it is given.How to Draft One
1
Give the assistant the assignment
Paste in the handout, or point the assistant at the handout repository. If the assistant is connected
to Pawtograder’s MCP server,
get_handout_files and get_grader_files return the handout and solution
repositories. See AI assistance for helping students for the setup. Anything
the assistant reads is sent to its model provider, and get_grader_files sends your solution
repository, so skip it if your course does not allow solutions to be shared with an outside service.2
Ask for options in Pawtograder's YAML
For example:
3
Paste the YAML into the rubric editor
Open Configure Rubric, select the Grading Review tab, switch to YAML source, replace the
contents with the assistant’s YAML, and click Save. The editor shows Rubric Saved when the
rubric is stored.
4
Check the rubric before grading
Confirm that the Grading Rubric Points Summary reports the total you expect. The rubric and the
autograder together should add up to the assignment’s max points, so if the autograder awards points,
the rubric covers only the rest. Then grade three or four real submissions with the rubric. Wherever you
hesitate between two options, rewrite their labels so the difference is something you can point to in
the submission.
Example
This rubric is in the shape to ask for. It is worth 20 points and can be pasted in as it stands.Troubleshooting
- The editor marks an error on a criteria. Every criteria needs
is_deduction_only. See How Points Are Computed. - The editor rejects a check’s options. A check needs at least two options, or none.
- Save fails with “Point values cannot be negative”. Use positive points and pick a scoring mode for deductions instead.
- The points summary shows 0 for the rubric. The checks have
points: 0. See Note on Options vs Base Points.
Editing Rubrics
When editing rubrics, Pawtograder protects against accidental data loss. If you attempt to leave the rubric editor with unsaved changes, a confirmation dialog appears warning you that your changes will be lost. You can choose to stay and save, or discard the changes and leave. This protection applies when:- Navigating away via links within the application
- Using browser back/forward buttons
- Closing the browser tab or window
- Refreshing the page
- Using the mobile navigation dropdown
How This Renders in the Hand-Grader
- Global checks appear as radio buttons (when
max_checks_per_submission = 1) or checkboxes (otherwise). - Checks with
data.optionsrender a choice list. The selected option’s label is prefixed to the comment. - Annotation checks are applied via right-click on a code line or by selecting an artifact.
- Checks linked to specific artifacts display a clickable link to navigate directly to that artifact.
- For each criteria, the sidebar shows:
- Additive:
earned / total_points - Subtractive:
remaining / total_points
- Additive:
Regrade Requests
Students can request regrades on rubric items where they believe the grading was incorrect. Regrade requests are automatically prevented on self-review rubric items, since those are completed by the students themselves.Split-Rubric Group Grading
For group assignments, you can apply per-student adjustments when using split-rubric grading. This allows you to assign different scores or provide individualized feedback to group members while maintaining the efficiency of group grading workflows.Editing Rubric Scores After Completion
TAs and instructors can edit or delete rubric scores even after completing a review. This allows for:- Correcting mistakes in grading
- Adjusting scores based on regrade requests
- Updating feedback after discussion with students
- Modifying annotations or comments
How to Edit After Completion
- Navigate to the completed review
- Click on any rubric score or annotation
- Make your changes (modify points, update comments, or delete checks)
- Changes are saved automatically
- The review remains marked as complete unless you explicitly mark it incomplete
Editing completed reviews does not automatically notify students. If grades have been released, you may need to communicate changes to affected students.
Notes and Tips
-
Keep criteria focused and keep check names short; longer explanation should go in
description. -
Use
max_annotationsto prevent over-counting nitpicks. - Prefer options when the same check has graded tiers (e.g., Complete/Partial/Incomplete).
-
Use
student_visibilityto separate internal notes from student-facing feedback. - Grade override notes are now properly displayed in the grading interface.
- Rubric scores are automatically recalculated when check scores change, ensuring the gradebook stays up-to-date.
- Save frequently to avoid losing work, especially for complex rubrics.
Field Reference: Required vs Optional (with defaults)
Rubric- Mandatory:
name,parts - Optional:
description
- Mandatory:
name,criteria - Optional:
id,description,data
- Mandatory:
name,checks,is_deduction_only(schema-required; usefalseunless the criteria is “Penalty only”) - Optional:
id,description,data,is_additive(default:false),total_points(default:0),min_checks_per_submission,max_checks_per_submission
- Mandatory:
name,is_annotation(boolean),is_required(boolean),is_comment_required(boolean),points(number; on an “Award per check” criteria withdata.options, set it to the highest option’s points, or the points summary counts the check as 0) - Optional:
id,description,file,artifact,annotation_target(file|artifact; default behavior in UI isfilewhen omitted for annotations),max_annotations,data(see below),student_visibility(default:always)
data.options (for multiple-choice checks)
- Mandatory per option:
label,points - Optional per option:
description - Notes: Must define at least two options (single-option checks are rejected by the editor). When options are present, the selected option’s
pointsreplace the basepointsfor that check when applied.
- Criteria without
is_additiveare treated as subtractive (deduction) criteria. - Criteria without
total_pointsdefault to0(i.e., no contribution unless checks add points in additive mode). - Check
student_visibilitydefaults toalways. - Annotation
annotation_targetdefaults tofilein the UI if omitted. min_checks_per_submissionandmax_checks_per_submissionare optional; when not set, graders are not constrained by count.
Grading Comment Restrictions After Release
Once a review has been released to students, grading comments are restricted to maintain grading integrity:- Graders can only edit or delete their own comments after a review is released
- Instructors retain full editing privileges and can modify any comment, even after release
- This prevents graders from modifying feedback after students have seen it, while allowing instructors to make corrections if needed
Check References (Cross-Rubric Context)
What It Is
A check on one rubric can reference a check on another rubric. During grading, any applied feedback (comments/points) from the referenced check(s) is shown inline under the current check as “Related Feedback from Other Reviews”. This gives graders context from, e.g., self-review or prior review rounds.Scoring and Visibility
- References are informational only. They do not contribute points to the current rubric’s criteria and do not alter the score computation.
- Referenced feedback is surfaced to graders in grading mode. Student visibility continues to follow each original check’s own
student_visibilityand release state within its source review.
How to Configure
- Navigate to the rubric preview/editor for an assignment.
- For the check you want to augment, click “Add Reference”.
- Search/select a check from other rubrics (the current rubric’s checks are excluded).
- Save. The relationship is stored so that, when grading, the referenced feedback appears under the referencing check.
Good Use Cases
- Show a student’s self-review evidence next to the corresponding grading check.
- Pull in meta-grading notes when doing final pass reviews.
Individual and Assign-to-Student Grading Modes
For group submissions, rubric parts can be configured with special grading modes to assess individual contributions:Individual Grading Mode
When a rubric part hasis_individual_grading: true, that part is graded separately for each group member. The grading interface displays the part once per student, and graders apply checks individually for each team member.
Use cases:
- Individual code quality assessments in group projects
- Personal reflection questions
- Individual contribution evaluations
Assign-to-Student Mode
When a rubric part hasis_assign_to_student: true, the grader selects which student the part applies to (or skips it if not applicable).
Use cases:
- Role-specific responsibilities (e.g., “Team Lead” duties)
- Optional bonus sections that only some students completed
- Targeted feedback for specific contributions
Scoring with Individual/Assign-to-Student Parts
When a rubric includes individual or assign-to-student parts:- Shared score: Points from regular (whole-group) rubric parts, autograder, and tweak
- Individual score: Points from individual-grading and assign-to-student parts for each student
- Total per student: Shared + individual, capped to assignment maximum if configured
Important Notes
- A part cannot be both
is_individual_gradingandis_assign_to_student(mutually exclusive) - Individual grading parts require every rubric-linked comment to name the student it applies to
- Assign-to-student parts are skipped in completion validation when not assigned to any student
- Students only see their own individual scores, not their teammates’ scores