Fix mouse exploration on Windows Terminal#20450
Conversation
|
Hi, Can someone test this on a touchscreen computer to make sure the fix also applies with that input modality? Thanks. |
|
I didn't want to test by building a self-signed launcher and installing it, so I used the following app module code for testing: # scratchpad\appModules\windowsterminal.py
"""Windows Terminal appModule"""
import appModuleHandler
import winUser
class AppModule(appModuleHandler.AppModule):
def isGoodUIAWindow(self, hwnd) -> bool:
if winUser.getClassName(hwnd) == "CASCADIA_HOSTING_WINDOW_CLASS":
return True
return super().isGoodUIAWindow(hwnd)The conclusion is that touch input can correctly read the text on the terminal. Furthermore, strangely enough, unlike the mouse, touch input can read the text line by line. Additionally, the following COM error is raised when touching the tab title text control: |
|
The error above may be related to WinUI 3; additionally, it is impossible to navigate to the minimize, maximize, and close buttons via touch exploration. |
|
I don't get any exception when hovering over a tab title with the mouse. The name of the Close/Maximize/Minimize buttons is not reported with or without this patch, but the Narrator also fails here so maybe not something we can fix on the client side. |
| # Windows Terminal's UIA text provider reports the paragraph, page and document units | ||
| # as spanning the entire terminal buffer. |
There was a problem hiding this comment.
Yeah, probably because UIA text ranges should fall back to the next-largest unit (i.e. if word isn't defined, it falls back to line). I don't think this is a bug.
CC @carlos-zamora.
There was a problem hiding this comment.
Correct. We fall back to the next-largest unit. We currently support these units: character, word, line, document.
Some relevant code (we do this a few times in that file): https://github.com/microsoft/terminal/blob/66ac36e7a47dd132267c5ce1f204f62da33647a9/src/types/UiaTextRangeBase.cpp#L1037-L1052
| # This file may be used under the terms of the GNU General Public License, version 2 or later, as modified by the NVDA license. | ||
| # For full terms and any additional permissions, see the NVDA license file: https://github.com/nvaccess/nvda/blob/master/copying.txt | ||
|
|
||
| """App module for the Windows Terminal application (wt.exe / WindowsTerminal.exe).""" |
There was a problem hiding this comment.
can this file be cased as WindowsTerminal if that's what Windows uses for the exe name
| class _WinTerminalUIATextInfo(UIATextInfo): | ||
| """A TextInfo for Windows Terminal that constrains mouse tracking to a single line.""" | ||
|
|
||
| def _get_unit_mouseChunk(self): |
FYI: Windows Terminal is built using WinUI 2 and XAML Islands, not WinUI 3. I forget why exactly we replaced the minimize, maximize, and close buttons, but the code for that in the Terminal repo is here: https://github.com/microsoft/terminal/blob/main/src/cascadia/TerminalApp/MinMaxCloseControl.xaml I'm noticing that they are marked as |
6 participants
Link to issue number:
Closes #20448
Summary of the issue:
Mouse exploration doesn't work on Windows Terminal due to the window hierarchy.
Description of user facing changes:
Hovering the mouse over a Windows Terminal window now reports the text (the whole line by default). Controls of the tab bar and scrollbars also work.
Description of developer facing changes:
Adds a dedicated app module for this application as well as a new TextRange class to make sure hovering over a window doesn't try to report the entire text buffer.
Description of development approach:
Testing strategy:
Opened a Windows Terminal window and moved the mouse around. Also tried with an instance with a large buffer.
Known issues with pull request:
None
Code Review Checklist: