Skip to content

Add Azure Pipelines incremental CodeQL scanning template (overlay + diff-informed)#23

Draft
Copilot wants to merge 2 commits into
mainfrom
copilot/create-sample-cli-usage-codeql
Draft

Add Azure Pipelines incremental CodeQL scanning template (overlay + diff-informed)#23
Copilot wants to merge 2 commits into
mainfrom
copilot/create-sample-cli-usage-codeql

Conversation

Copilot AI commented Jun 9, 2026

Copy link
Copy Markdown

Implements the incremental CodeQL analysis pattern for ADO pipelines, combining overlay analysis (reuse cached base DB, re-extract only changed files) with diff-informed analysis (restrict alert output to PR-changed lines). Can cut PR scan times by up to 10× on large codebases.

New file: Azure Pipelines/Azure-Pipelines-template-linux-incremental.yml

Based on Azure-Pipelines-template-linux.yml. Single pipeline YAML with two stages selected automatically by trigger:

Trigger Stage Behaviour
Push to default branch OverlayBaseScan Full scan with --overlay-base; publishes DB + OID snapshot as pipeline artifact
Pull request IncrementalPRScan Overlay + diff-informed incremental scan; falls back to full scan if no artifact found

Stage 1 — OverlayBaseScan (push to default branch)

  • database init --overlay-base --db-clusterdatabase finalize (extraction for interpreted languages)
  • Records Git OIDs for all tracked files (snapshot used by PRs to detect changed files)
  • database run-queries without --expect-discarded-cache to preserve the overlay query cache
  • database cleanup --cache-cleanup=overlay before publishing
  • Publishes DB + OIDs as codeql-overlay-base-<language> pipeline artifact; uploads full SARIF

Stage 2 — IncrementalPRScan (pull request)

  • Downloads latest overlay-base artifact from the target branch (continueOnError: true for fallback)
  • Changed-file computation: compares current Git OIDs against the base snapshot → overlay-changes.json
  • Diff-informed extension pack: parses git diff --unified=0 merge_base..HEAD hunk headers → generates qlpack.yml + pr-diff-range.yml feeding the restrictAlertsTo extensible predicate
  • database init --overlay-changes + database finalize (re-extracts changed files only)
  • database run-queries --additional-packs --extension-packs=my-ci/pr-diff-range
  • database interpret-results --sarif-run-property=incrementalMode=overlay,diff-informed
  • CI-side SARIF filter: drops results whose locations don't intersect the diff ranges (required since restrictAlertsTo is permissive, not guaranteed)
  • Fallback path (database create + database analyze) when no overlay artifact is available

Requirements: CodeQL CLI Bundle 2.23.9+ (JS/Python/Ruby) or 2.23.8+ (other languages); Git 2.38.0+; build-mode: none only (compiled languages except Go not supported by overlay).

Updated: Azure Pipelines/README.md

Added entry for the new template with a trigger/behaviour summary table and requirements.

Copilot AI changed the title [WIP] Create a sample showing CLI usage of CodeQL incremental scanning Add Azure Pipelines incremental CodeQL scanning template (overlay + diff-informed) Jun 9, 2026
Copilot AI requested a review from felickz June 9, 2026 18:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create a sample that shows CLI usage of CodeQL incremental scanning ( diff informed + overlay)

2 participants