Skip to content

desktop::uia live test rejects visually-hidden 1x1 elements far off-screen (fails locally on a web page) #52

Closed

Description

@blockmandev

What fails

desktop::uia::tests::live::the_focused_window_can_be_walked fails on a Windows desktop whose focused window is a web page with a screen-reader-only element:

panicked at src/desktop/uia.rs:1178:17
element Element { id: 153, depth: 12, role: Text,
  name: "To view keyboard shortcuts, press question mark View keyboard shortcuts",
  rect: Rect { x: 0, y: -16079, width: 1, height: 1 }, .. }
is nowhere near the desktop Rect { x: 0, y: 0, width: 1920, height: 1080 }

It fails 3 times out of 3 with that window focused. CI passes, because its runners have no such window.

Why

  • Pages hide text from sight but keep it for screen readers by drawing it 1×1 far off-screen (the usual "visually hidden" CSS).
  • UI Automation reports those rectangles faithfully.
  • The assertion at uia.rs:1178-1184 expects every element within one desktop's size of the desktop (meant to catch garbage coordinates, uia.rs:1174-1177). A real, valid element breaks that.

Two questions to decide

  1. The test. Should the bounds check skip 1×1 (or zero-area-ish) elements, or only apply to elements large enough to be clicked? The intent, catching garbage pixels, holds either way.
  2. The walk. Should the walk itself drop elements that are entirely off every screen? Today it keeps them (controls past the edge of a small screen are kept deliberately). A 1×1 element 16 000 px away cannot be clicked, but its text may still be useful context for the model.

Activity

  1. added a commit that references this issue on Oct 22, 2026
  2. blockmandev commented on Oct 22, 2026

    @blockmandev
    MemberAuthor

    The test half is fixed in #55 (merged). The live walk test now checks that coordinates are screen pixels at all (within 2^24 px), not how close they are to the screen, so a visually-hidden 1×1 element or a long page's below-the-fold content no longer fails it. A unit test pins the element from this issue.

    Still open here: question 2. Should the walk itself drop elements that are entirely off every screen? Their text may still help the model, but they can't be clicked.

  3. blockmandev commented on Oct 23, 2026

    @blockmandev
    MemberAuthor

    Solved in #55. The live test now accepts real off-screen elements (like this page's hidden 1×1 text) and still rejects corrupt coordinates. A unit test covers this exact element; CI is green on all three OSes. The remaining design question moved to #56.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions