Add Standard Code Test assessment JSON guide by jairovelasquez · Pull Request #469 · codio/knowledge

jairovelasquez · 2026-06-09T15:03:28Z

Summary by CodeRabbit

Documentation
- Added a comprehensive guide for authoring Standard Code Test assessments in Codio, including configuration options, test cases, and practical examples.

coderabbitai · 2026-06-09T15:04:06Z

Walkthrough

This PR adds a new documentation guide for authoring "Standard Code Test" assessment JSON files in Codio. The guide covers storage location (.guides/assessments/), filename conventions, complete JSON schema structure, field references, configuration options, test case format, and multiple authoring examples. A navigation entry is added to the Guides toctree for discoverability.

Changes

Assessment documentation

Layer / File(s)	Summary
Standard Code Test assessment JSON guide and integration `source/instructors/authoring/guides/standard_code_test_assessment_json.rst`, `source/instructors/authoring/guides/intro.rst`	New guide comprehensively documents `code-output-compare` assessment JSON structure, including storage rules, schema fields (type, taskId, source, options, metadata, sequence), comparison settings (timeout, case/whitespace/newline handling, substring matching), test case format with arguments/input/output/feedback, display options for rationale and expected answers, and usage examples for CLI arguments, stdin, and substring matching. Guide can be embedded in Markdown via `{Check It!

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

codio/knowledge#465: Adds another assessment-authoring guide (multiple_choice_assessment_json) to the same Guides section and uses the same toctree integration pattern.
codio/knowledge#435: Updates Standard Code Test authoring documentation that this guide directly complements with JSON schema details.

Suggested reviewers

shajason

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately summarizes the main change: adding a new comprehensive guide for Standard Code Test assessment JSON documentation.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

jairovelasquez · 2026-06-09T15:04:22Z

@coderabbitai review

coderabbitai · 2026-06-09T15:04:27Z

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

coderabbitai

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@source/instructors/authoring/guides/standard_code_test_assessment_json.rst`:
- Around line 430-431: Replace the verbose learningObjectives value (key
"learningObjectives") with a concise plain-language sentence that removes the
unexplained acronym "SWBAT" (e.g., "Students will calculate simple interest
using basic arithmetic"), and while you're editing the adjacent "guidance" value
keep it brief and consistent in tone (e.g., "Implements simple interest
calculation with basic arithmetic and formatting") so both fields read clearly
and match the guide's concise style.
- Around line 67-74: The guide currently shows inconsistent embed token spacing
— `{Check It!|assessment}` vs elsewhere `{Text |assessment}` — so pick one
canonical token format (recommend no space before the pipe, e.g. `{Check
It!|assessment}`) and update this file's example text and the surrounding
guidance to explicitly state the chosen canonical form and that authors must use
it exactly (mention the exact token pattern and that there should be no space
before the `|`), ensuring any other occurrences in related docs are aligned to
the same format.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 8bb6575c-f273-4f95-8352-9df518f6f669

📥 Commits

Reviewing files that changed from the base of the PR and between 25242e5 and 819e74e.

📒 Files selected for processing (2)

source/instructors/authoring/guides/intro.rst
source/instructors/authoring/guides/standard_code_test_assessment_json.rst

LolaValente · 2026-06-09T17:51:36Z

+
+.. _standard-code-test-assessment-json:
+
+Standard Code Test assessment JSON


Assessment should be capitalized

LolaValente · 2026-06-09T17:52:04Z

+In the assessment JSON, Standard Code Tests use the assessment type
+``code-output-compare``.
+
+File location and naming


Change to "File Location and Naming"

LolaValente · 2026-06-09T17:53:31Z

+
+   {Check It!|assessment}(code-output-compare-1075712907)
+
+Use ``{Check It!|assessment}`` exactly as shown, with no spaces before the closing brace.


Maybe add this in an important box?

LolaValente · 2026-06-09T17:54:22Z

+     }
+   }
+
+Field reference


Capitalize r in Reference

LolaValente · 2026-06-09T17:55:32Z

+Field reference
+---------------
+
+Top-level fields


Change to "Top-Level Fields"

LolaValente · 2026-06-09T18:00:26Z

+     }
+   }
+
+Standard input


Capitalize "i"

LolaValente · 2026-06-09T18:01:00Z

+     }
+   }
+
+Matching a substring


Capitalize "s"

LolaValente · 2026-06-09T18:01:39Z

+Set ``matchSubstring`` to ``false`` when the learner's output should match the
+expected output more strictly, subject to the other comparison options.
+
+Assessment files and opened files


Change to "Assessment Files and Opened Files"

LolaValente · 2026-06-09T18:02:29Z

+     ]
+   }
+
+Guide embedding example


Change to "Guide Embedding Example"

LolaValente · 2026-06-09T18:02:48Z

+
+   {Check It!|assessment}(code-output-compare-165558969)
+
+Authoring notes


Capitalize "n"

Changing to approved

SCT JSON

819e74e

coderabbitai Bot reviewed Jun 9, 2026

View reviewed changes

Comment thread source/instructors/authoring/guides/standard_code_test_assessment_json.rst

Comment thread source/instructors/authoring/guides/standard_code_test_assessment_json.rst

jairovelasquez changed the title ~~SCT JSON~~ Add Standard Code Test assessment JSON guide Jun 9, 2026

LolaValente previously requested changes Jun 9, 2026

View reviewed changes

LolaValente approved these changes Jun 10, 2026

View reviewed changes


		.. _standard-code-test-assessment-json:

		Standard Code Test assessment JSON


		{Check It!\|assessment}(code-output-compare-1075712907)

		Use ``{Check It!\|assessment}`` exactly as shown, with no spaces before the closing brace.


		{Check It!\|assessment}(code-output-compare-165558969)

		Authoring notes

Conversation

jairovelasquez commented Jun 9, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Possibly related PRs

Suggested reviewers

Uh oh!

jairovelasquez commented Jun 9, 2026

Uh oh!

coderabbitai Bot commented Jun 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jairovelasquez commented Jun 9, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading

coderabbitai Bot commented Jun 9, 2026 •

edited

Loading