Adding end to end tests for the guides and samples#351
Merged
Conversation
- Create SampleTestHelpers.psm1 shared PowerShell module with assertion, logging, CLI invocation, and MSIX packaging helpers - Add self-contained test.ps1 for each sample: cpp-app, dotnet-app, electron, flutter-app, rust-app, tauri-app, wpf-app - Add test-samples.yml GitHub Actions workflow with matrix strategy running 7 samples in parallel, triggered by Build and Package workflow - Add scripts/test-samples.ps1 local orchestrator for running tests locally with pass/fail summary - Update AGENTS.md with sample testing conventions and instructions Each test validates: prerequisites -> build -> package MSIX -> verify output. Tests run without elevation. electron-winml skipped (requires ML models). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Rewrite all 7 sample test.ps1 scripts to follow a guide-first approach: - Phase 1: From-scratch guide workflow in a temp directory (scaffold project, winapp init, build, cert generate, cert info, pack MSIX) - Phase 2: Quick build of existing sample code to verify freshness Add new packaging-cli guide test (samples/packaging-cli/test.ps1): - Tests winapp manifest generate, cert generate, cert info, pack, sign - Validates the docs/guides/packaging-cli.md workflow end-to-end Update shared module with new helpers: - New-TempTestDirectory / Remove-TempTestDirectory for temp dir lifecycle - Assert-WinappInitOutput for verifying winapp init creates expected files - Assert-CertInfo for verifying winapp cert info output Add packaging-cli to CI workflow matrix (8 parallel jobs total). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Fix [switch]$Verbose conflict with PowerShell common parameter in all test scripts and orchestrator (use [CmdletBinding()] instead) - Fix $PSScriptRoot in module functions pointing to wrong directory (pass -SampleDir $PSScriptRoot from callers) - Fix Assert-WinappInitOutput defaults (switches default to $false) - Remove -ExpectWinappYaml for .NET and --setup-sdks=none projects (.NET uses .csproj, Rust/Tauri with no SDKs skip winapp.yaml) - Fix dotnet/wpf tests to explicitly call winapp pack after build (auto-packaging MSBuild targets are optional, not added by init) - Find .exe output directory recursively (handles RID subdirectories) - Fix winapp sign syntax: positional args, not --cert flag - Simplify electron test to sample freshness check only (from-scratch Electron guide workflow is covered by E2E test in test-e2e-electron.ps1) - Rename workflow and headings to 'Sample & Guide' - Update CI matrix: packaging-cli needs Node.js, electron no longer needs .NET Validated locally: packaging-cli, dotnet-app, rust-app, electron, wpf-app Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- Replace 8 raw PowerShell test.ps1 files with Pester test.Tests.ps1 files - Simplify SampleTestHelpers.psm1 from ~460 to ~180 lines (Pester handles assertions/reporting) - Use BeforeDiscovery + BeforeAll dual-phase pattern for prerequisite skip logic - Update orchestrator (scripts/test-samples.ps1) as thin Pester wrapper with comma-split support - Update CI workflow for Invoke-Pester with JUnit XML test result reporting - Update AGENTS.md with Pester conventions for sample tests - Fix winapp cert info positional arg syntax in wpf-app test - Resolve WinappPath to absolute in orchestrator before passing to containers All 8 sample tests validated locally: packaging-cli: 10/10, electron: 7/7, dotnet-app: 15/15, rust-app: 15/15, wpf-app: 10/10, tauri-app: 17/17, cpp-app: 8 skipped (no cmake), flutter-app: 7 skipped (no flutter) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…l sample tests - Replace all Invoke-Expression 'winapp ...' calls with Invoke-WinappCommand -Arguments in dotnet-app, packaging-cli, rust-app, and tauri-app tests. This ensures tests work in environments where winapp isn't on PATH by using the helper's fallback chain (npx -> dotnet run -> PATH). - Fix flutter-app to use -Arguments named parameter consistently. - Remove duplicate \ assignments in dotnet-app and rust-app. - Update AGENTS.md to document Context-level -Skip as acceptable when BeforeAll has prerequisite-dependent setup. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Build Metrics ReportBinary Sizes
Test Results✅ 722 passed out of 722 tests in 381.2s (-140.6s vs. baseline) Test Coverage❌ 20.6% line coverage, 34.6% branch coverage · ✅ no change vs. baseline CLI Startup Time44ms median (x64, Updated 2026-04-17 20:24:10 UTC · commit |
Migrate the full Electron guide workflow from scripts/test-e2e-electron.ps1 into samples/electron/test.Tests.ps1 as Phase 1, making electron consistent with all other sample tests (Phase 1: from-scratch guide + Phase 2: sample freshness check). Changes: - Rewrite electron test.Tests.ps1 with Phase 1 covering: Electron app creation, winapp init, C++/C# addon creation and build, debug identity, Electron packaging, certificate generation, and MSIX packaging. - Remove e2e-test job from build-package.yml (now covered by test-samples.yml). - Delete scripts/test-e2e-electron.ps1 (replaced by Pester test). - Add .NET SDK setup for electron in test-samples.yml (needed for C# addon). Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The workflow_run trigger only works for workflow files on the default branch, so test-samples.yml won't trigger until merged. Adding pull_request trigger lets sample tests run as proper PR checks. For PR events, a build job produces the npm-package artifact first. The test-sample jobs then download it, same as workflow_run/dispatch. The build job is skipped for non-PR triggers since the artifact comes from the Build and Package workflow instead. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nsure nuget package is referenced correctly in the pipeline
steps, add winapp run
Member
Author
|
@copilot resolve the merge conflicts in this pull request |
Contributor
Resolved by merging latest |
Member
Author
|
@copilot resolve the merge conflicts in this pull request |
Co-authored-by: nmetulev <711864+nmetulev@users.noreply.github.com>
Contributor
Resolved in |
…Package.appxmanifest winapp init and manifest generate now output Package.appxmanifest instead of appxmanifest.xml. Update all Phase 1 test assertions and --manifest arguments to match the new filename. Phase 2 (existing sample) references are unchanged since those samples still have appxmanifest.xml committed. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…nifest The tauri test copies the existing sample (which has appxmanifest.xml) before running winapp init. Since init with --use-defaults preserves existing manifests, the file remains appxmanifest.xml rather than being regenerated as Package.appxmanifest. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 28 out of 29 changed files in this pull request and generated 4 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Resolve-WinappCliPath: include artifacts\ in default candidates and pick newest .tgz by LastWriteTime - Invoke-WinappCommand: prefer winapp on PATH over dotnet run against repo source so tests exercise the packaged CLI; opt back into dotnet via WINAPP_TEST_USE_DOTNET=1 - test-samples.yml: drop workflow_run trigger to avoid duplicate runs per PR; build job now also runs on workflow_dispatch so the workflow is self-contained; simplify artifact downloads - AGENTS.md: fix local example to use .\artifacts\ where package-npm.ps1 emits the tarball Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Adds end-to-end Pester tests for all 8 guides and samples, running automatically on every PR. These tests exercise the core CLI workflows (
winapp init,winapp run,winapp pack,winapp cert,winapp node create-addon, etc.) across all supported frameworks. This gives us regression coverage so that changes to the CLI — like modifyingwinapp init, package resolution, or the addon template — are validated against real-world workflows before merging. The three bug fixes in this PR were discovered directly by this test coverage.Each sample has a self-contained
test.Tests.ps1that validates the guide workflow from scratch (Phase 1) and verifies the existing sample code still builds (Phase 2).Test Coverage Matrix
winapp initwinapp restorewinapp runwinapp packwinapp signwinapp cert generatewinapp cert infowinapp manifest generatemanifest add-aliascreate-debug-identityadd-electron-debug-identitynode create-addon(C++)node create-addon(C#)Infrastructure
.github/workflows/test-samples.ymlwith matrix strategy (8 parallel jobs)samples/SampleTestHelpers.psm1scripts/test-samples.ps1for local executionType of Change
Checklist
Notes