> ## Documentation Index
> Fetch the complete documentation index at: https://docs.pawtograder.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Test Insights

> Aggregate autograder failures across the class to find systemic issues and bulk-regrade affected submissions

Test Insights aggregates autograder results across every active submission in an assignment so you can spot a flaky test, an ambiguous spec, or an off-by-one in your reference solution without clicking through individual submissions. From any error group you can view affected students, copy their emails, pin the error globally, or launch a bulk regrade against a chosen grader version.

Open an assignment's management page and select the **Test Insights** tab.

## Overview

The overview tab summarizes how the class is doing against the autograder:

* **Total Submissions** and how many have graded results.
* **Tests Analyzed** — the number of distinct test cases in the suite.
* **Avg Pass Rate** across all tests.
* **Hardest Test** — the test with the lowest pass rate.
* **Overall Score Distribution** — a bar chart bucketing students by score.
* **Test Performance Overview** — a sortable table of every test with difficulty badge, pass rate, average score, total attempts, passing count, and failing count.
* **Submissions to Full Marks** — for tests where students can converge on full credit, the count of students with and without full marks and the average/median number of submissions it took.

## Common Errors Explorer

The explorer groups failing test results by normalized output, so the same `AssertionError: expected 5 but was 4` from 30 different students appears as a single row instead of 30.

Each group shows the test name and part, an error signature, the number of affected submissions, and the average score on that test. Expanding a group reveals sample error outputs and a button to load and copy the email addresses of every affected student.

### Filtering and sorting

Use the filter panel above the list to narrow results:

* **Test Name** and **Test Part** — restrict to a specific test.
* **Min Occurrences** — hide error patterns affecting fewer than N submissions (default 2).
* **Search Output** — substring search across test name, signature, and sample outputs.
* **Sort By** — occurrence count, average score, or test name, ascending or descending.

### Actions on an error group

Each expanded group exposes four actions:

* **AI Analyze** — copies a pre-built prompt to your clipboard with context about the error and affected submissions. Paste it into the AI assistant of your choice to get analysis or draft a discussion post.
* **View Submissions** — opens the autograder page with the affected submissions pre-selected.
* **Regrade Submissions** — opens the regrade dialog (see below).
* **Create Error Pin** — links this error pattern to a discussion thread. When a student hits the same error in the future, they see a callout pointing at your explanation. Pins can be scoped to this assignment or made global (class-wide). See [Discussion error pins](/staff/discussion) for details.

## Bulk regrade dialog

The regrade dialog targets every submission in the selected error group. It lets you pick which grader version to run:

* **Select a commit from main branch** — choose from the last 50 commits to the autograder repo's `main` branch (defaults to the latest).
* **Custom SHA** — enter any valid SHA from the grader repository, including branches or tags.

The dialog also shows the affected student emails (collapsed by default, with a copy-all button) so you can notify students before kicking off the run.

### Auto-promote

The **Auto-promote new result to official** checkbox is on by default. With it enabled, the new grader result replaces the student's current official score as soon as regrading completes.

<Tip>
  Disable auto-promote when you're not yet sure the new grader version is correct. The new scores will land as "What-if" results that you can review and promote manually.
</Tip>

<Warning>
  Regrading is asynchronous and may take several minutes depending on submission count. Track progress on the Workflow Runs page.
</Warning>

## Typical workflow

<Steps>
  <Step title="Scan the explorer after the deadline">
    Sort by occurrence count and look for groups affecting many students. A pattern of 10+ failures on the same test usually means either a widespread misconception or a broken test.
  </Step>

  <Step title="Investigate the pattern">
    Expand the group and read the sample outputs. Use **AI Analyze** to get a starting analysis, or check the test source if it looks like a grader bug.
  </Step>

  <Step title="Take action">
    For a student-side issue, copy the affected emails and post a clarification on the discussion board, then create an Error Pin linking to it. For a grader bug, push a fix to the autograder repo and use **Regrade Submissions** with the new commit.
  </Step>

  <Step title="Promote when confident">
    If you ran the regrade with auto-promote off, review the What-if results before promoting them to official.
  </Step>
</Steps>