Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.pawtograder.com/llms.txt

Use this file to discover all available pages before exploring further.

​
Configuring Autograders

This documentation is under construction. Check back soon for detailed autograder configuration guides.

​
Summary of Autograding Architecture

The autograder system uses an “overlay grader” architecture that allows flexible grading configurations.

​
Empty Submission Detection

Pawtograder automatically detects empty submissions to help identify students who may need assistance. A submission is considered empty if:
  • The repository contains no files beyond the initial starter code
  • No meaningful changes have been made to the starter files
  • The submission appears to be unmodified from the template
Empty submissions are flagged in the grading interface, allowing instructors to quickly identify students who may not have started the assignment or encountered issues with their repository setup. Detailed documentation coming soon.

​
Artifact Rendering

Autograders can generate artifacts (files) that are displayed to students and graders alongside submission results. The system supports rendering various file formats:

​
Supported Artifact Types

  • Plaintext files (.txt, .log, etc.): Displayed with syntax highlighting and line numbers
  • Markdown files (.md): Rendered as formatted HTML with proper styling
  • Binary files: Available for download but not rendered inline

​
Configuring Artifacts

To include artifacts in your autograder output:
  1. Generate the artifact files in your grading script
  2. Place them in the expected output directory
  3. Configure the artifact list in your assignment settings
  4. Artifacts will automatically appear in the submission view

​
Artifact Annotations

Graders can apply rubric checks as annotations directly to artifacts:
  • Use annotation_target: artifact in your rubric check configuration
  • Specify the artifact filename in the artifact field
  • Graders can select text or sections of the artifact to attach feedback
  • This is useful for grading written reports, documentation, or generated output files
See the Rubrics documentation for more details on configuring artifact annotations.

​
Artifacts

Autograders can produce artifacts that are displayed to students and graders. Artifacts are files or content generated during the grading process.

​
Supported Artifact Types

Plaintext artifacts
  • Display plain text content in a formatted code block
  • Useful for test output, logs, or simple reports
  • Automatically formatted with monospace font
Markdown artifacts
  • Render formatted markdown content
  • Support headings, lists, code blocks, and other markdown features
  • Ideal for detailed feedback reports or documentation
Binary artifacts
  • Support for other file types (images, PDFs, etc.)
  • Displayed with appropriate viewers
To create artifacts, your autograder should output files that Pawtograder can detect and display. The system automatically detects the file type and renders it appropriately.

​
AI-Powered Hints

Pawtograder supports AI-powered hints for autograder test failures. Instructors can configure tests to provide LLM-generated hints to help students understand and fix errors.

​
Supported AI Providers

  • OpenAI - GPT models (default: gpt-4o-mini)
  • Azure OpenAI - Azure-hosted OpenAI models
  • Anthropic - Claude models
  • OpenRouter - Access to multiple AI providers through a unified API
Configure the AI provider in your test configuration using the llm field with provider, model, temperature, and max_tokens options.

​
AI-Powered Hints for Students

Pawtograder supports AI-powered hints for failing tests using LLM providers. When a student fails a test, they can request a hint that provides guidance without giving away the solution.

​
Supported LLM Providers

Configure AI hints using one of these providers:
  • OpenAI - GPT models (default: gpt-4o-mini)
  • Azure OpenAI - Azure-hosted GPT models
  • Anthropic - Claude models
  • OpenRouter - Access to multiple LLM providers through a unified API

​
Configuration

Set up LLM hints in your test configuration by adding an llm field to the test’s extra_data:

​
AI-Powered Feedback with Feedbot

Feedbot provides AI-generated feedback on student submissions using large language models (LLMs). It can analyze code, provide suggestions, and help students understand their mistakes.

​
Supported LLM Providers

Feedbot supports multiple LLM providers:
  • OpenAI: GPT-3.5, GPT-4, and other OpenAI models
  • OpenRouter: Access to a wide variety of models including Claude, Llama, and others through a unified API

​
Configuring OpenRouter

To use OpenRouter with Feedbot:
  1. Navigate to the assignment’s autograder settings
  2. Select Feedbot as the feedback provider
  3. Choose OpenRouter as the LLM provider
  4. Enter your OpenRouter API key
  5. Select the model you want to use (e.g., anthropic/claude-3-sonnet, meta-llama/llama-3-70b)
  6. Configure feedback parameters:
    • Temperature (creativity level)
    • Max tokens (response length)
    • System prompt (instructions for the AI)
OpenRouter provides access to many models at competitive pricing, making it easier to experiment with different AI models for providing feedback to students. Pawtograder includes Feedbot, an AI-powered tool that can provide automated feedback on student submissions. Feedbot supports multiple AI providers:
  • OpenAI: GPT models for generating feedback
  • OpenRouter: Access to a wide variety of AI models through a unified API
Instructors can configure Feedbot to use their preferred AI provider and model to generate personalized feedback for students based on their code submissions and test results.

​
Test Insights and Error Explorer

Pawtograder provides powerful tools to help you understand and address common errors across student submissions:

​
Error Explorer

The error explorer groups similar test failures and build errors across all student submissions, making it easy to:
  • Identify patterns: See which errors affect multiple students
  • View affected students: Click on any error group to see the list of students experiencing that issue
  • Copy student emails: Quickly copy email addresses of affected students for bulk communication
  • Launch regrades: Start a regrade workflow directly from the error explorer

​
Improved Regrade Workflow

When you need to regrade submissions after fixing a test or updating the autograder:
  1. Launch from error explorer: Click the regrade button from any error group
  2. Preselected submissions: Students affected by that error are automatically selected
  3. Choose commits or manual SHA: Select which commits to regrade in a dialog
  4. Auto-promote option: Automatically promote the new submission if it scores higher
This streamlined workflow makes it much faster to address issues that affect multiple students.

​
AI-Powered Hints

Autograder test results can include AI-generated hints to help students debug their code. Configure LLM hints by adding an llm object to the test’s extra_data: 
{
  "llm": {
    "type": "v1",
"prompt": "You are a helpful CS tutor. Guide the student to fix their code without giving away the solution.",
    "provider": "openrouter",
    "model": "openai/gpt-4o-mini",
    "account": "default",

"prompt": "Help the student understand why their test failed...",
    "model": "gpt-4o-mini",
    "provider": "openai",
    "temperature": 0.85,
    "max_tokens": 500
  }
}

​
Provider-Specific Configuration

​
OpenRouter

OpenRouter provides access to multiple LLM providers through a single API. Set the OPENROUTER_API_KEY environment variable or use account-specific keys with OPENROUTER_API_KEY_{account}. Model names use the format provider/model, for example:
  • openai/gpt-4o-mini
  • anthropic/claude-3-haiku
  • google/gemini-pro

​
OpenAI

Set OPENAI_API_KEY or use account-specific keys with OPENAI_API_KEY_{account}.

​
Azure OpenAI

Requires AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_KEY (or account-specific variants).

​
Anthropic

Set ANTHROPIC_API_KEY or use account-specific keys with ANTHROPIC_API_KEY_{account}.

​
Supported Providers

  • OpenAI: Use "provider": "openai" with models like gpt-4o-mini, gpt-4o
  • Azure OpenAI: Use "provider": "azure" with your Azure deployment
  • Anthropic: Use "provider": "anthropic" with models like claude-3-haiku, claude-3-sonnet
  • OpenRouter: Use "provider": "openrouter" with any OpenRouter-supported model (e.g., openai/gpt-4o-mini, anthropic/claude-3-haiku)

​
Configuration

Set environment variables for API keys:
  • OPENAI_API_KEY or OPENAI_API_KEY_{account} for OpenAI
  • AZURE_OPENAI_KEY or AZURE_OPENAI_KEY_{account} for Azure
  • ANTHROPIC_API_KEY or ANTHROPIC_API_KEY_{account} for Anthropic
  • OPENROUTER_API_KEY or OPENROUTER_API_KEY_{account} for OpenRouter
Use the account field in your LLM configuration to specify which account-specific key to use.

​
Test Insights and Error Analysis

Pawtograder provides powerful tools for analyzing common test failures across student submissions:
  • Error Explorer: View common error patterns grouped by test name and error signature
  • Affected Students: See which submissions are impacted by each error pattern
  • Regrade Workflow: Launch regrades directly from the error explorer with automatic submission selection
  • Email Export: Copy affected students’ emails for communication
These tools help you quickly identify whether test failures indicate a problem with the assignment specification or common student mistakes.

​
Submission File Support

Pawtograder supports both text and binary files in student submissions:

​
Text Files

Text files (code, markdown, configuration files, etc.) are displayed with syntax highlighting in the submission viewer. Students and graders can view the file contents directly in the browser.

​
Binary Files

Binary files (images, PDFs, compiled executables, etc.) are supported in submissions. These files:
  • Are stored securely in the submission
  • Can be downloaded by students and graders
  • Display a download button in the file viewer
  • Show file metadata (name, size, type)

​
Markdown Preview

Markdown files (.md, .markdown) in submissions are automatically rendered with a live preview. This allows students and graders to see formatted documentation, README files, and written responses without needing to download the file. The markdown preview supports:
  • Standard markdown syntax (headings, lists, links, code blocks)
  • Inline code and code blocks with syntax highlighting
  • Images and links
  • Tables and blockquotes

​
Test Insights and Error Explorer

Pawtograder provides powerful tools to help you understand and debug common student errors:

​
Error Explorer

View grouped test failures across all submissions to identify patterns and common issues. The error explorer allows you to:
  • See which students are affected by specific errors
  • Copy affected students’ email addresses for communication
  • Launch regrading workflows directly from error groups

​
Regrade Workflow

The improved regrade workflow streamlines the process of re-running autograders:
  1. Launch from Error Explorer: Start a regrade directly from an error group
  2. Preselect Submissions: Submissions are automatically selected on the autograder page
  3. Choose Commits: Select specific commits or enter a manual SHA in the regrade dialog
  4. Auto-promote: Optionally promote the new submission as the active submission
This workflow includes conflict detection and clearer validation for solo office-hours requests.

​
Test Insights and Error Analysis

Pawtograder provides powerful tools to help instructors understand and respond to common student errors:

​
Error Explorer

The error explorer groups similar test failures together, allowing you to:
  • View all students affected by a particular error pattern
  • Copy affected students’ email addresses for bulk communication
  • Launch regrading workflows directly from error groups
  • Identify systemic issues in assignments or test cases

​
Regrade Workflow

When you need to regrade submissions, you can:
  1. Launch regrade from the error explorer
  2. Preselect affected submissions on the autograder page
  3. Choose specific commits or enter a manual SHA in the regrade dialog
  4. Auto-promote the new submission if desired
This streamlined workflow makes it easy to respond to discovered issues or updated test cases.

​
Empty Submission Detection

The autograder automatically detects empty submissions to help identify students who may need assistance. Empty submissions are flagged when:
  • No meaningful code changes are present
  • Only starter code or template files exist
  • The submission contains no substantive work
This feature helps instructors quickly identify students who may be struggling to get started on an assignment.

​
Empty Submission Detection

Pawtograder automatically detects empty submissions where students have not made meaningful changes to the starter code. When an empty submission is detected:
  • The submission is flagged with a warning indicator
  • Students are notified that their submission appears to be empty
  • Instructors can see which submissions are empty in the grading interface
This helps identify students who may need assistance or who accidentally submitted without completing their work.

​
Test Insights and Error Explorer

Pawtograder provides powerful tools to help instructors understand and address common test failures across student submissions.

​
Error Explorer

The error explorer groups similar test failures together, making it easy to:
  • Identify patterns in student errors
  • See which students are affected by each error type
  • View and copy affected students’ email addresses for bulk communication
  • Pin error groups globally to highlight common issues

​
Regrade Workflow

When you need to regrade submissions after fixing test issues:
  1. Launch a regrade directly from the error explorer
  2. Submissions are automatically preselected on the autograder page
  3. Choose to regrade from specific commits or enter a manual SHA
  4. Option to auto-promote the new submission as the active submission
This streamlined workflow makes it easy to regrade multiple students who encountered the same issue, without manually selecting each submission. Pawtograder automatically detects empty submissions to help identify students who may need assistance. When a student creates a submission without any meaningful code changes, the system flags it as potentially empty. This helps instructors:
  • Identify students who may be struggling to get started
  • Catch accidental submissions before grading
  • Provide early intervention for students who need help
Empty submissions are detected based on the content of files configured for the assignment. The system analyzes whether the submission contains substantive changes from the assignment template or starter code.

​
Rerunning Autograders

Instructors can rerun the autograder on student submissions when needed. The enhanced rerun feature allows you to:
  • Reuse the same submission: Rerun the autograder on the existing submission without creating a new one
  • Select grader version: Choose which version of the grader to use for the rerun, allowing you to test grader updates or revert to previous versions
  • Batch operations: Rerun autograders on multiple submissions at once
This is useful when:
  • You’ve fixed a bug in the autograder
  • You want to apply updated test cases to existing submissions
  • You need to verify grading consistency across different grader versions
You can rerun the autograder on existing submissions without creating new submissions. This is useful when:
  • You’ve fixed a bug in your grading script
  • You want to update test cases and regrade existing work
  • You need to regrade after adjusting point values
When rerunning an autograder:
  1. Navigate to the assignment’s autograder page
  2. Select the submissions you want to regrade
  3. Choose which version of the grader to use:
    • Latest commit: Use the most recent version of your grader repository
    • Specific commit: Select a particular commit SHA for consistent grading
    • Auto-promote: Automatically promote the new autograder results if they improve the student’s score
The rerun reuses the same submission rather than creating a new one, preserving the original submission timestamp and history.

​
Test Insights and Regrade Workflow

The test insights dashboard integrates with the autograder to streamline regrading:
  1. View Error Groups: See which test cases are failing and how many students are affected
  2. Launch Regrade: Start a regrade directly from an error group
  3. Preselect Submissions: Affected submissions are automatically selected on the autograder page
  4. Choose Grader Version: Select commits or enter a manual SHA in the regrade dialog
  5. Auto-promote Results: Optionally promote new results automatically if scores improve
This workflow makes it easy to identify issues, fix them, and regrade affected students efficiently.

​
Test Insights and Error Explorer

Pawtograder provides powerful test insights to help instructors identify and address common student errors:
  • Error Explorer: View grouped test failures across all submissions to identify patterns
  • Regrade Workflow: Launch regrading directly from the error explorer with preselected submissions
  • Email Export: View and copy affected students’ emails from error groups for targeted communication
  • Flexible Regrade Options: Choose specific commits or manual SHA in a regrade dialog with auto-promote functionality
These tools help instructors quickly identify widespread issues and efficiently regrade submissions when autograder configurations are updated.

​
Rerunning the Autograder

You can rerun the autograder on existing submissions to apply updated grading logic or fix issues. The enhanced rerun feature provides more control over the regrading process.

​
Rerun Options

When rerunning the autograder, you can:
  1. Reuse existing submissions: The autograder reruns on the same submission without creating a new one
  2. Select grader version: Choose which version of the grader to use for the rerun
    • Select from recent commits in your grader repository
    • Enter a specific commit SHA manually
    • Use the current active grader version

​
How to Rerun

To rerun the autograder on submissions:
  1. Navigate to the assignment’s autograder page
  2. Select the submissions you want to regrade
  3. Click the “Rerun Autograder” button
  4. In the regrade dialog:
    • Choose a grader version from the commit list, or
    • Enter a specific commit SHA
    • Optionally enable “Auto-promote” to set this version as the active grader

​
Auto-Promote Feature

The auto-promote option automatically promotes the selected grader version to be the active grader for all future submissions after the rerun completes successfully. This is useful when:
  • You’ve fixed a bug in the grader and want to apply it going forward
  • You’ve updated test cases and want all new submissions to use them
  • You’re confident the new grader version should be the default
Rerunning the autograder will replace the existing autograder results for the selected submissions. Make sure you’ve tested your grader changes before applying them to student submissions.
Instructors can rerun the autograder on student submissions when needed. The enhanced rerun feature allows you to:
  • Rerun the autograder on the same submission (without creating a new submission)
  • Select which version of the grader to use for the rerun
  • Preserve the original submission timestamp and metadata
This is useful when you’ve fixed a bug in the autograder, want to test a new grading configuration, or need to regrade submissions after updating test cases.

​
Quickstart: Java and Python

Build system integration guides for common programming languages. Detailed documentation coming soon.

​
AI-Assisted Grading with Feedbot

Feedbot provides AI-powered feedback on student submissions using large language models (LLMs). It can analyze code quality, provide suggestions, and generate personalized feedback at scale.

​
Supported LLM Providers

Feedbot supports multiple LLM providers for flexibility and cost optimization:

​
OpenAI

  • Models: GPT-4, GPT-4 Turbo, GPT-3.5 Turbo
  • Best for: High-quality feedback, complex code analysis
  • Configuration: Requires OpenAI API key

​
Anthropic Claude

  • Models: Claude 3 Opus, Claude 3 Sonnet, Claude 3 Haiku
  • Best for: Detailed explanations, educational feedback
  • Configuration: Requires Anthropic API key

​
OpenRouter

OpenRouter provides access to multiple LLM providers through a single API, including:
  • OpenAI models (GPT-4, GPT-3.5)
  • Anthropic models (Claude 3 family)
  • Open-source models (Llama, Mistral, etc.)
  • Specialized models for code analysis
Benefits of OpenRouter:
  • Single API key for multiple providers
  • Automatic fallback if primary model is unavailable
  • Cost optimization through model selection
  • Access to latest models without separate integrations
Configuration:
  1. Sign up for an OpenRouter account at openrouter.ai
  2. Generate an API key from your OpenRouter dashboard
  3. In Pawtograder assignment settings, select OpenRouter as the LLM provider
  4. Enter your OpenRouter API key
  5. Choose which model to use from the OpenRouter catalog
  6. Configure feedback parameters (temperature, max tokens, etc.)

​
Configuring Feedbot for an Assignment

  1. Navigate to the assignment settings page
  2. Scroll to the AI Grading section
  3. Enable Feedbot
  4. Select your LLM provider (OpenAI, Anthropic, or OpenRouter)
  5. Enter the appropriate API key
  6. Choose the model to use
  7. Configure feedback settings:
    • Feedback style (concise, detailed, encouraging)
    • Focus areas (code quality, style, logic, testing)
    • Maximum feedback length
    • Language and tone preferences

​
Feedbot Feedback Types

Code Quality Analysis:
  • Identifies code smells and anti-patterns
  • Suggests refactoring opportunities
  • Evaluates naming conventions and readability
Logic and Correctness:
  • Reviews algorithm efficiency
  • Identifies potential bugs or edge cases
  • Suggests alternative approaches
Style and Best Practices:
  • Checks adherence to language conventions
  • Recommends documentation improvements
  • Evaluates code organization
Custom Prompts: Create custom prompts for assignment-specific feedback:
  1. In the Feedbot configuration, click Add Custom Prompt
  2. Write a prompt describing what to evaluate
  3. Specify which files or code sections to analyze
  4. Set priority and weight for this feedback type

​
Viewing Feedbot Results

Students see Feedbot feedback on their submission page:
  • Inline comments on specific code lines
  • Summary feedback at the submission level
  • Suggestions for improvement
  • Links to relevant documentation or resources
Instructors can:
  • Review all Feedbot feedback before releasing to students
  • Edit or override AI-generated feedback
  • Disable Feedbot for specific submissions
  • Track Feedbot usage and costs

​
Best Practices

  • Start with conservative feedback settings and adjust based on results
  • Review Feedbot output on sample submissions before enabling for all students
  • Combine Feedbot with traditional autograding for comprehensive assessment
  • Use custom prompts to align feedback with course learning objectives
  • Monitor API costs and set usage limits if needed

​
AI-Powered Feedback with Feedbot

Pawtograder includes Feedbot, an AI-powered feedback system that can provide automated feedback on student submissions. Feedbot integrates with various AI providers to generate contextual feedback based on rubric criteria and submission content.

​
Supported AI Providers

Feedbot supports multiple AI providers for flexibility and cost optimization:
  • OpenAI: GPT-4 and other OpenAI models
  • OpenRouter: Access to multiple AI models through a single API, including:
    • Anthropic Claude models
    • Google Gemini models
    • Meta Llama models
    • And many others available through OpenRouter’s unified interface

​
Configuring Feedbot

To enable Feedbot for your course:
  1. Navigate to course settings
  2. Select “AI Feedback Configuration”
  3. Choose your preferred AI provider (OpenAI or OpenRouter)
  4. Enter your API credentials
  5. Configure feedback generation settings
  6. Enable Feedbot for specific assignments

​
Using OpenRouter

OpenRouter provides access to a wide variety of AI models through a single API endpoint, making it easy to:
  • Compare different models for your use case
  • Switch between providers without code changes
  • Access models that may be more cost-effective for your needs
  • Use specialized models for specific feedback types
To use OpenRouter with Feedbot, select “OpenRouter” as your provider and provide your OpenRouter API key in the configuration settings.

​
Rerunning the Autograder

When you need to regrade submissions (for example, after fixing a bug in the autograder), you can rerun the autograder on existing submissions:

​
Rerun Options

When rerunning the autograder, you can:
  • Reuse the same submission: The autograder runs again on the existing submission without creating a new one
  • Select grader version: Choose which version of the grader to use for the rerun
    • Use the latest grader commit
    • Select a specific commit SHA
    • Choose from recent grader versions

​
Launching a Rerun

You can launch autograder reruns from multiple places:
  1. From the assignment page: Select submissions and click “Rerun Autograder”
  2. From test insights: Launch reruns directly from error groups (see Test Insights Dashboard)
  3. From individual submissions: Rerun the autograder for a single submission
When launching from test insights, affected submissions are automatically preselected, streamlining the regrade workflow.

​
Test Insights and Error Explorer

Pawtograder provides tools to help you understand common errors across student submissions:

​
Error Explorer

View and analyze test failures grouped by error type. The error explorer helps you:
  • Identify patterns in student errors
  • See which students are affected by specific issues
  • Copy affected students’ email addresses for bulk communication
  • Pin error groups globally to highlight common issues

​
Regrade Workflow

Launch regrading directly from the error explorer:
  1. Select submissions from the error explorer
  2. Navigate to the autograder page with submissions preselected
  3. Choose commits or enter a manual SHA in the regrade dialog
  4. Optionally auto-promote the new submission as the active submission
This streamlined workflow makes it easy to regrade submissions after fixing autograder issues or updating test cases.

​
Test Insights Dashboard

The test insights dashboard helps you track student performance and identify common issues across submissions. Access it from the assignment page to view:

​
Error Explorer

View and analyze test failures across all student submissions:
  • Error grouping: Automatically groups similar test failures together
  • Affected students: See which students are experiencing each error
  • Email export: Copy affected students’ email addresses for bulk communication
  • Global error pins: Pin important errors to track them across assignments

​
Regrade Workflow

Launch targeted regrades directly from the error explorer:
  1. Select errors: Choose which error groups need regrading
  2. Preselect submissions: The autograder page automatically preselects affected submissions
  3. Regrade dialog: Choose between:
    • Specific commits from the repository history
    • Manual SHA entry for precise version control
  4. Auto-promote: Optionally promote the regrade results automatically
This workflow streamlines the process of fixing grading issues and re-evaluating student work without manual submission selection.

​
Rerunning the Autograder

Instructors can rerun the autograder on student submissions when needed. This is useful for:
  • Testing autograder changes before applying them to all submissions
  • Debugging grading issues
  • Updating scores after fixing autograder bugs

​
Enhanced Rerun Features

When rerunning the autograder, you can:
  • Reuse the same submission: The rerun updates the existing submission rather than creating a new one, preserving the submission history and student’s active submission status
  • Select grader version: Choose which version of the grader to use for the rerun, allowing you to test new grader versions or revert to previous versions
This flexibility ensures you can iterate on your autograder configuration without disrupting student submissions or creating confusion with multiple submission entries.