feat: add support for utf-8 transcoding and unicode console mode for exec commands#62
Open
Vladyslav Nikonov (vnikonov-devolutions) wants to merge 2 commits into
Open
feat: add support for utf-8 transcoding and unicode console mode for exec commands#62Vladyslav Nikonov (vnikonov-devolutions) wants to merge 2 commits into
Vladyslav Nikonov (vnikonov-devolutions) wants to merge 2 commits into
Conversation
…batch, winps, pwsh, process
Vladyslav Nikonov (vnikonov-devolutions)
requested a review
from Benoît Cortier (CBenoit)
There was a problem hiding this comment.
Pull request overview
This PR bumps NOW-PROTO to v1.6 and introduces encoding-control support for exec commands, including OEM→UTF-8 transcoding controls and a Unicode console mode signal via a new capability flag.
Changes:
- Add new exec flags for RAW_ENCODING, UNICODE_CONSOLE, and PROCESS_ENCODING_UTF8 across Rust + .NET message models/builders.
- Add
NOW_CAP_EXEC_UNICODE_CONSOLEcapability + bump protocol version to 1.6 (Rust/.NET + docs/tests). - Expand the .NET CLI test app to exercise the new flags and help validate encoding behavior.
Reviewed changes
Copilot reviewed 18 out of 18 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| rust/now-proto-testsuite/tests/proto/channel.rs | Updates roundtrip expectations for protocol minor version bump (1.5 → 1.6). |
| rust/now-proto-pdu/src/exec/win_ps.rs | Adds WinPS exec flags + builder/query helpers for raw encoding and unicode console mode. |
| rust/now-proto-pdu/src/exec/pwsh.rs | Adds Pwsh exec flags + builder/query helpers (reusing WinPS flag set). |
| rust/now-proto-pdu/src/exec/process.rs | Adds process flag + helpers for enabling OEM→UTF-8 transcoding. |
| rust/now-proto-pdu/src/exec/batch.rs | Adds batch flags + helpers for raw encoding and unicode console mode. |
| rust/now-proto-pdu/src/channel/capset.rs | Adds capability flag + bumps NowProtoVersion::CURRENT to 1.6. |
| dotnet/Devolutions.NowProto/src/NowProtoVersion.cs | Bumps .NET protocol version constant to 1.6. |
| dotnet/Devolutions.NowProto/src/Messages/NowMsgExecWinPs.cs | Adds WinPS message flags + builder toggles + exposed properties. |
| dotnet/Devolutions.NowProto/src/Messages/NowMsgExecPwsh.cs | Adds Pwsh message flags + builder toggles + exposed properties. |
| dotnet/Devolutions.NowProto/src/Messages/NowMsgExecProcess.cs | Adds EncodingUtf8 flag support to serialization/deserialization + builder. |
| dotnet/Devolutions.NowProto/src/Messages/NowMsgExecBatch.cs | Adds batch raw/unicode-console flags support to serialization/deserialization + builder. |
| dotnet/Devolutions.NowProto/src/Capabilities/NowCapabilityExec.cs | Adds UnicodeConsole capability and includes it in All. |
| dotnet/Devolutions.NowClient/src/ExecWinPsParams.cs | Exposes new WinPS params to set raw/unicode-console and maps them to message flags. |
| dotnet/Devolutions.NowClient/src/ExecPwshParams.cs | Exposes new Pwsh params to set raw/unicode-console and maps them to message flags. |
| dotnet/Devolutions.NowClient/src/ExecProcessParams.cs | Exposes new process param EncodingUtf8 and maps it to the message flag. |
| dotnet/Devolutions.NowClient/src/ExecBatchParams.cs | Exposes new batch params to set raw/unicode-console and maps them to message flags. |
| dotnet/Devolutions.NowClient.Cli/Program.cs | Adds interactive commands and output capture to test cmd/winps/pwsh encoding flags. |
| docs/NOW-spec.md | Updates spec to 1.6 and documents the new flags/capability + encoding behavior notes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| /// | ||
| /// NOW-PROTO: NOW_CAP_EXEC_IO_REDIRECTION | ||
| const IO_REDIRECTION = 0x1000; | ||
| /// Set if host supports encoding control flags for exec commands\n /// (RAW_ENCODING, UNICODE_CONSOLE, and ENCODING_UTF8 flags). |
Comment on lines
7
to
8
|
|
||
|
|
Comment on lines
+1140
to
+1142
| **Encoding behavior (Pwsh):** PowerShell 7 (pwsh) defaults to UTF-8 output natively, so no OEM transcoding is performed by default. | ||
| `NOW_EXEC_FLAG_PS_RAW_ENCODING` has no additional effect since pwsh already outputs UTF-8. | ||
| Set `NOW_EXEC_FLAG_PS_UNICODE_CONSOLE` to inject the encoding setup snippet for explicit UTF-8 console configuration. |
|
|
||
| **Encoding behavior:** By default, batch (cmd.exe) output is transcoded from OEM encoding to UTF-8, since cmd.exe uses the system OEM code page. | ||
| Set `NOW_EXEC_FLAG_BATCH_RAW_ENCODING` to disable transcoding and pass raw bytes through unchanged. | ||
| Set `NOW_EXEC_FLAG_BATCH_UNICODE_CONSOLE` to inject `@chcp 65001 > nul` and pass UTF-8 streams directly without transcoding. |
Contributor
Author
|
Copilot please open PR for issues found after review; also please fix dotnet tests (tests fail after default capabilities were changed) |
Contributor
|
Vladyslav Nikonov (@vnikonov-devolutions) I've opened a new pull request, #63, to work on those changes. Once the pull request is ready, I'll request review from you. |
…s/comments (#63) This follow-up addresses post-review issues from the UTF-8 transcoding / Unicode console support work. It fixes failing .NET capset snapshot tests after the protocol default/version change and resolves documentation/comment formatting gaps flagged in review. - **.NET tests: update capset wire snapshots for protocol v1.6** - Updated expected encoded bytes in channel capset tests to match `NowProtoVersion.Current = 1.6`. - **Spec: clarify combined RAW + UNICODE behavior** - Clarified behavior for batch and WinPS when both flags are set: - Agent **MUST** enable Unicode console mode for process startup. - Agent **MUST NOT** perform stdout/stderr transcoding. - Returned output is raw process output bytes; clients **MUST NOT** assume a specific output encoding. - Fixed inline-code formatting for `NOW_EXEC_FLAG_PS_RAW_ENCODING`. - **Rust docs: fix malformed doc comment rendering** - Removed literal `\n` sequence and corrected indentation in `NowExecCapsetFlags::UNICODE_CONSOLE` docs so rustdoc renders correctly. - **C# formatting: remove accidental blank lines** - Cleaned extra blank lines in `NowProtoVersion.cs` near `Current`. ```csharp // MsgChannel capset expected version bytes updated from 1.5 to 1.6 // before: ... 0x01, 0x00, 0x05, 0x00, ... // after: ... 0x01, 0x00, 0x06, 0x00, ... ``` --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: vnikonov-devolutions <246051166+vnikonov-devolutions@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR updates now proto to v1.6, adding the following:
now-protonow requires agent to implement server-side OEM code page transcoding to UTF-8.processexecution mode, raw (no-transcoding) is still default, but transcoding could be enabled byNOW_EXEC_FLAG_PROCESS_ENCODING_UTF8flag instead, asprocessexecution is more advanced use case usually needed for bit-bit output, so no explicit transcoding is provided.NOW_CAP_EXEC_UNICODE_CONSOLEcapability to signify that new flags are supported and the redirected streams are indeed correct utf-8.