Test Insights

Test Insights aggregates autograder results across every active submission in an assignment so you can spot a flaky test, an ambiguous spec, or an off-by-one in your reference solution without clicking through individual submissions. From any error group you can view affected students, copy their emails, pin the error globally, or launch a bulk regrade against a chosen grader version. Open an assignment’s management page and select the Test Insights tab.

​

Overview

The overview tab summarizes how the class is doing against the autograder:

• Total Submissions and how many have graded results.
• Tests Analyzed — the number of distinct test cases in the suite.
• Avg Pass Rate across all tests.
• Hardest Test — the test with the lowest pass rate.
• Overall Score Distribution — a bar chart bucketing students by score.
• Test Performance Overview — a sortable table of every test with difficulty badge, pass rate, average score, total attempts, passing count, and failing count.
• Submissions to Full Marks — for tests where students can converge on full credit, the count of students with and without full marks and the average/median number of submissions it took.

​

Common Errors Explorer

The explorer groups failing test results by normalized output, so the same AssertionError: expected 5 but was 4 from 30 different students appears as a single row instead of 30. Each group shows the test name and part, an error signature, the number of affected submissions, and the average score on that test. Expanding a group reveals sample error outputs and a button to load and copy the email addresses of every affected student.

​

Filtering and sorting

Use the filter panel above the list to narrow results:

• Test Name and Test Part — restrict to a specific test.
• Min Occurrences — hide error patterns affecting fewer than N submissions (default 2).
• Search Output — substring search across test name, signature, and sample outputs.
• Sort By — occurrence count, average score, or test name, ascending or descending.

​

Actions on an error group

Each expanded group exposes four actions:

• AI Analyze — copies a pre-built prompt to your clipboard with context about the error and affected submissions. Paste it into the AI assistant of your choice to get analysis or draft a discussion post.
• View Submissions — opens the autograder page with the affected submissions pre-selected.
• Regrade Submissions — opens the regrade dialog (see below).
• Create Error Pin — links this error pattern to a discussion thread. When a student hits the same error in the future, they see a callout pointing at your explanation. Pins can be scoped to this assignment or made global (class-wide). See Discussion error pins for details.

​

Bulk regrade dialog

The regrade dialog targets every submission in the selected error group. It lets you pick which grader version to run:

• Select a commit from main branch — choose from the last 50 commits to the autograder repo’s main branch (defaults to the latest).
• Custom SHA — enter any valid SHA from the grader repository, including branches or tags.

The dialog also shows the affected student emails (collapsed by default, with a copy-all button) so you can notify students before kicking off the run.

​

Auto-promote

The Auto-promote new result to official checkbox is on by default. With it enabled, the new grader result replaces the student’s current official score as soon as regrading completes.

​

Typical workflow