PM-5015 - ai only review mode#43
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
Adds support for finalizing "AI_ONLY" challenges, in which AI decision scores replace human review summations for ranking and awarding winners. Phase-closure and scheduler logic are extended so that the new "AI Review" phase is treated equivalently to "AI Screening" for workflow-completion gating, and a new ReviewService.getAiDecisionSummaries query feeds AI scores to ChallengeCompletionService for completion or cancellation.
Changes:
- Introduces
AI_REVIEW_PHASE_NAMEconstant and recognizes the "AI Review" phase in autopilot, scheduler, and completion paths. - Implements
finalizeAiOnlyChallengeinChallengeCompletionService(ranking, tie-breaking, cancellation, payouts) andgetAiDecisionSummariesinReviewService(latest non-PENDING decision per submission). - Extends
SchedulerServiceto defer closing the AI Review phase while AI workflows are still in progress, and to clean up retry attempts for that phase.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| src/autopilot/constants/review.constants.ts | Adds AI_REVIEW_PHASE_NAME constant. |
| src/autopilot/services/autopilot.service.ts | Treats AI Review like AI Screening for AI workflow completion and phase closure. |
| src/autopilot/services/scheduler.service.ts | Blocks AI Review close until AI workflows finish and clears retry attempts; reuses AI-screening defer/key helpers. |
| src/autopilot/services/challenge-completion.service.ts | Detects AI_ONLY challenges by phase name and adds finalizeAiOnlyChallenge using AI decision scores. |
| src/review/review.service.ts | Adds getAiDecisionSummaries raw SQL query to retrieve latest AI decisions per submission. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+827
to
+847
| if (inProgressAiWorkflows > 0) { | ||
| await this.deferAiScreeningPhaseClosure( | ||
| data, | ||
| inProgressAiWorkflows, | ||
| `${inProgressAiWorkflows} in-progress AI workflow run(s) detected`, | ||
| ); | ||
| return; | ||
| } | ||
| } catch (error) { | ||
| const err = error as Error; | ||
| this.logger.error( | ||
| `[AI REVIEW LATE] Unable to verify AI workflow readiness for phase ${data.phaseId} on challenge ${data.challengeId}: ${err.message}`, | ||
| err.stack, | ||
| ); | ||
|
|
||
| await this.deferAiScreeningPhaseClosure( | ||
| data, | ||
| undefined, | ||
| 'unable to verify AI workflow readiness', | ||
| ); | ||
| return; |
Comment on lines
+557
to
+563
| const isAiOnly = (challenge.phases ?? []).some( | ||
| (p) => p.name === AI_REVIEW_PHASE_NAME, | ||
| ); | ||
|
|
||
| if (isAiOnly) { | ||
| return this.finalizeAiOnlyChallenge(challengeId, challenge); | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://topcoder.atlassian.net/browse/PM-5015
This pull request introduces support for finalizing "AI_ONLY" challenges, where AI decisions are used in place of human review summations to determine winners. The changes add the ability to recognize and process the new
AI Reviewphase, ensuring that phase closure and challenge completion logic correctly handle this mode. Additionally, the pull request implements logic to rank and award winners based on AI decision scores and provides a new service method for retrieving AI decision summaries.Support for AI_ONLY challenge finalization:
AI_REVIEW_PHASE_NAMEconstant and updated all relevant imports to recognize the new phase name. [1] [2] [3] [4]ChallengeCompletionService, added detection for AI_ONLY challenges and implementedfinalizeAiOnlyChallenge, which finalizes challenges based on AI decision scores, handles tie-breaking, and manages challenge cancellation when necessary. [1] [2]Phase closure and workflow handling:
AutopilotServiceto treat theAI Reviewphase as equivalent to theAI Screeningphase for AI_ONLY challenges, ensuring phase closure logic works for both. [1] [2]SchedulerService, added logic to block closure of theAI Reviewphase until all configured AI workflows are complete, mirroring the screening phase's behavior. Also ensures retry attempts are managed for the new phase. [1] [2] [3]AI decision summary retrieval:
getAiDecisionSummariesmethod toReviewService, which retrieves and processes the latest AI decisions for all contest submissions, filtering and marking passing submissions appropriately for use in AI_ONLY challenge finalization.