Skip to content

Commit dd29324

Browse files
committed
demo with cloud tracing, more steps for post-execution verification
1 parent 1628af2 commit dd29324

5 files changed

Lines changed: 246 additions & 5 deletions

File tree

demo/CHANGELOG.md

Lines changed: 87 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Changelog - Predicate Secure Demo
2+
3+
## [2024-02-25] - Interactive Click Functionality
4+
5+
### Added
6+
- **Interactive clicking**: Demo now finds and clicks the "Learn more" link on example.com using semantic element query
7+
- **Semantic element finding**: Uses `find()` function from predicate SDK with DSL query `"role=link text~'Learn more'"`
8+
- **Post-click verification**: Automatically verifies URL contains "example-domains" after clicking via LLM-generated verifications
9+
10+
### Implementation Details
11+
12+
#### New Methods
13+
1. **`_find_and_click_link(snapshot, link_text)`**
14+
- Uses semantic query to find links by text
15+
- Falls back gracefully if link not found
16+
- Wraps click in authorized action pattern with verification
17+
18+
2. **`_click_element(element)`**
19+
- Clicks element using Playwright selector
20+
- Falls back to coordinate-based clicking if selector fails
21+
22+
#### Enhanced Methods
23+
3. **`_authorized_action()` now returns result**
24+
- Returns the executor result for use in subsequent actions
25+
- Enables capturing snapshot for element finding
26+
27+
4. **`_run_browser_task()` updated**
28+
- Step 1: Navigate to example.com
29+
- Step 2: Take snapshot (with overlay)
30+
- Step 3: Find and click "Learn more" link
31+
- Post-verification checks URL contains "example-domains"
32+
33+
### Policy Changes
34+
- Added `click` action to authorization policy
35+
- Added `element#*` resource pattern for element ID-based clicks
36+
- Updated `allow-browser-click-safe-elements` rule
37+
38+
### Verification Flow
39+
When clicking the link, the demo:
40+
1. **Pre-execution authorization**: Checks click action is allowed by policy
41+
2. **Execute click**: Uses Playwright to click the element
42+
3. **Post-execution verification**: LLM generates verifications including:
43+
- URL changed from example.com
44+
- URL contains "example-domains"
45+
- Page content updated
46+
- Element interaction successful
47+
48+
### Visual Features
49+
- Snapshot overlay enabled (`show_overlay=True`)
50+
- Elements highlighted in browser during snapshot capture
51+
- Console shows element details (ID, role, clickability)
52+
53+
## [2024-02-25] - Cloud Tracing Integration
54+
55+
### Added
56+
- **Cloud tracing**: Upload authorization and verification events to Predicate Studio
57+
- **Run tracking**: Each demo run gets unique UUID and timestamp label
58+
- **Event emission**:
59+
- Authorization events (action, target, decision)
60+
- Verification events (predicates, reasoning, pass/fail)
61+
- **Studio integration**: View execution timeline at `https://studio.predicatesystems.dev/runs/{run_id}`
62+
63+
### Configuration
64+
- Automatic when `PREDICATE_API_KEY` is set in `.env`
65+
- Uses `create_tracer()` from predicate SDK
66+
- Blocking upload on cleanup to ensure events are sent
67+
68+
## [2024-02-24] - Initial Release
69+
70+
### Core Features
71+
- Pre-execution authorization via policy file
72+
- Post-execution verification via local LLM (Qwen 2.5 7B)
73+
- Apple Silicon MPS support via `device_map="auto"`
74+
- AsyncPredicateBrowser integration
75+
- Visual element overlay during snapshot capture
76+
77+
### Dependencies
78+
- `predicate-runtime==1.1.2` (browser automation)
79+
- `predicate-authority>=0.1.0` (authorization)
80+
- Qwen 2.5 7B Instruct (local LLM)
81+
- Rich console output
82+
83+
### Documentation
84+
- Quick start guide (5 minutes)
85+
- Full setup instructions
86+
- Sidecar setup guide (optional)
87+
- Architecture diagrams

demo/QUICKSTART.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -179,6 +179,8 @@ The demo executed a complete **pre-execution authorization + post-execution veri
179179
- `snapshot_changed()` - Check page loaded ✓
180180
4. **Snapshot**: Captured page elements with **visual overlay highlights**
181181
- Watch the browser window - you'll see colored boxes around detected DOM elements!
182+
5. **Cloud Tracing** (if API key set): Events uploaded to Predicate Studio ✓
183+
- View the execution timeline at: `https://studio.predicatesystems.dev/runs/{run_id}`
182184

183185
All checks passed → Action successful!
184186

demo/README.md

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -60,8 +60,9 @@ This demo showcases the complete **pre-execution authorization + post-execution
6060
- **Fail-Closed by Default**: All actions denied unless explicitly allowed by policy
6161
- **Dynamic Verification**: Local LLM generates verification assertions based on action context and page state
6262
- **Visual Element Overlay**: Watch the browser highlight detected DOM elements with colored boxes during snapshot
63+
- **Cloud Tracing**: When API key is provided, authorization and verification events are automatically traced to Predicate Studio
6364
- **Rich Console Output**: Beautiful terminal output with real-time progress indicators
64-
- **Audit Trail**: All authorization decisions and verification results logged
65+
- **Audit Trail**: All authorization decisions and verification results logged locally and to cloud (if API key provided)
6566

6667
## Prerequisites
6768

@@ -76,13 +77,23 @@ This demo showcases the complete **pre-execution authorization + post-execution
7677

7778
The demo works with **FREE TIER** (local browser extension only) by default. No API key needed!
7879

79-
If you have a Predicate API key for enhanced features:
80+
If you have a Predicate API key, you get enhanced features:
8081
```bash
8182
# In .env
8283
PREDICATE_API_KEY=your-api-key-here
8384
```
8485

85-
**Free tier is completely sufficient for this demo.** The demo works entirely offline after initial model download.
86+
**With API key, you get:**
87+
-**Cloud Tracing**: Authorization and verification events automatically uploaded to Predicate Studio
88+
-**Visual Timeline**: View execution flow, authorization decisions, and verification results in the Studio UI
89+
-**API-based Snapshots**: Faster and more reliable snapshot capture
90+
-**Run ID**: Each demo run gets a unique UUID for tracking in Studio
91+
92+
**Without API key (FREE TIER):**
93+
- ✅ Full demo functionality
94+
- ✅ Local browser extension
95+
- ✅ Offline operation (after model download)
96+
- ✅ Console logging of all events
8697

8798
### Required Packages
8899

demo/policies/browser_automation.yaml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ rules:
6060
actions:
6161
- "browser.click"
6262
- "browser.element.click"
63+
- "click" # Allow simple click action
6364
resources:
6465
# Allow clicks on common safe element types
6566
- "element:button[*"
@@ -69,6 +70,7 @@ rules:
6970
- "element:role=button[*"
7071
- "element:role=link[*"
7172
- "element:role=searchbox[*"
73+
- "element#*" # Allow clicks on elements by ID (from snapshot)
7274
conditions:
7375
required_labels:
7476
- "element_visible"

demo/secure_browser_demo.py

Lines changed: 141 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,9 @@
1313
import logging
1414
import os
1515
import sys
16+
import uuid
1617
from pathlib import Path
18+
from datetime import datetime
1719

1820
from dotenv import load_dotenv
1921
from rich.console import Console
@@ -68,6 +70,11 @@ def __init__(self):
6870
self.verifier = None
6971
self.secure_agent = None
7072
self.browser = None
73+
self.tracer = None
74+
75+
# Generate run ID for cloud tracing
76+
self.run_id = str(uuid.uuid4())
77+
self.run_label = f"predicate-secure-demo-{datetime.now().strftime('%Y%m%d-%H%M%S')}"
7178

7279
def _init_verifier(self):
7380
"""Initialize local LLM verifier."""
@@ -83,6 +90,39 @@ def _init_verifier(self):
8390
progress.update(task, completed=True)
8491
console.print("[green]✓[/green] Verifier initialized\n")
8592

93+
def _init_tracer(self):
94+
"""Initialize cloud tracer if API key is provided."""
95+
api_key = os.getenv("PREDICATE_API_KEY")
96+
if not api_key or self.tracer is not None:
97+
return
98+
99+
console.print("\n[bold cyan]Initializing Cloud Tracer...[/bold cyan]")
100+
101+
try:
102+
from predicate.tracer_factory import create_tracer
103+
104+
self.tracer = create_tracer(
105+
api_key=api_key,
106+
run_id=self.run_id,
107+
upload_trace=True,
108+
goal=f"[demo] {self.task_description}",
109+
agent_type="predicate-secure/demo",
110+
llm_model="Qwen/Qwen2.5-7B-Instruct",
111+
start_url=self.start_url,
112+
)
113+
114+
console.print("[green]✓[/green] Cloud tracer initialized")
115+
console.print(f" [dim]Run ID: {self.run_id}[/dim]")
116+
console.print(f" [dim]Run Label: {self.run_label}[/dim]")
117+
console.print(
118+
f" [dim]View trace in Predicate Studio: https://studio.predicatesystems.dev/runs/{self.run_id}[/dim]\n"
119+
)
120+
except Exception as e:
121+
logger.warning(f"Failed to initialize cloud tracer: {e}")
122+
console.print(
123+
f" [yellow]⚠[/yellow] Cloud tracer initialization failed: {e}\n"
124+
)
125+
86126
def _init_secure_agent(self):
87127
"""Initialize SecureAgent with predicate-authority integration."""
88128
if self.secure_agent is not None:
@@ -139,6 +179,7 @@ async def run_demo(self):
139179
try:
140180
# Step 1: Initialize components
141181
self._init_verifier()
182+
self._init_tracer() # Initialize cloud tracer if API key provided
142183
self._init_secure_agent()
143184

144185
# Step 2: Initialize browser (with authorization)
@@ -205,13 +246,16 @@ async def _run_browser_task(self):
205246
executor=lambda: self.browser.goto(self.start_url), # Returns coroutine
206247
)
207248

208-
# Action 2: Take snapshot
209-
await self._authorized_action(
249+
# Action 2: Take snapshot to find clickable elements
250+
snapshot = await self._authorized_action(
210251
action="snapshot",
211252
target="current_page",
212253
executor=lambda: self._take_snapshot(), # Returns coroutine
213254
)
214255

256+
# Action 3: Find and click the "Learn more" link using semantic query
257+
await self._find_and_click_link(snapshot, "Learn more")
258+
215259
console.print("\n[green]✓[/green] Task completed successfully\n")
216260

217261
async def _authorized_action(self, action: str, target: str, executor: callable):
@@ -231,6 +275,19 @@ async def _authorized_action(self, action: str, target: str, executor: callable)
231275
# For this demo, we'll simulate the authorization check
232276
authorized = self._check_authorization(action, target)
233277

278+
# Emit authorization event to cloud tracer
279+
if self.tracer:
280+
self.tracer.emit(
281+
"authorization",
282+
data={
283+
"action": action,
284+
"target": target,
285+
"principal": self.principal_id,
286+
"authorized": authorized,
287+
"policy_file": str(self.policy_file),
288+
},
289+
)
290+
234291
if not authorized:
235292
console.print(" [red]✗[/red] Action denied by policy")
236293
raise PermissionError(f"Action {action} denied by authorization policy")
@@ -279,12 +336,34 @@ async def _authorized_action(self, action: str, target: str, executor: callable)
279336
console.print(" [dim]Executing verifications...[/dim]")
280337
all_passed = self._execute_verifications(verification_plan)
281338

339+
# Emit verification event to cloud tracer
340+
if self.tracer:
341+
self.tracer.emit(
342+
"verification",
343+
data={
344+
"action": action,
345+
"target": target,
346+
"verifications": [
347+
{
348+
"predicate": v.predicate,
349+
"args": v.args,
350+
"passed": v.passed if hasattr(v, "passed") else None,
351+
}
352+
for v in verification_plan.verifications
353+
],
354+
"reasoning": verification_plan.reasoning,
355+
"all_passed": all_passed,
356+
},
357+
)
358+
282359
if all_passed:
283360
console.print(" [green]✓[/green] All verifications passed")
284361
else:
285362
console.print(" [red]✗[/red] Some verifications failed")
286363
raise AssertionError("Post-execution verification failed")
287364

365+
return result
366+
288367
def _check_authorization(self, action: str, target: str) -> bool:
289368
"""Check if action is authorized by policy.
290369
@@ -315,6 +394,54 @@ def _check_authorization(self, action: str, target: str) -> bool:
315394
# For other actions, default to allow for demo
316395
return True
317396

397+
async def _find_and_click_link(self, snapshot, link_text: str):
398+
"""Find a link by text using semantic query and click it.
399+
400+
This demonstrates using the predicate SDK's find() function for
401+
semantic element selection from snapshot.
402+
"""
403+
from predicate import find
404+
405+
console.print(f"\n[yellow]→[/yellow] Finding link with text: '{link_text}'")
406+
407+
# Use semantic query to find the link
408+
# The find() function returns the best match by importance
409+
element = find(snapshot, f"role=link text~'{link_text}'")
410+
411+
if not element:
412+
console.print(f" [yellow]⚠[/yellow] Link '{link_text}' not found, skipping click")
413+
return
414+
415+
console.print(f" [green]✓[/green] Found element: {element.text} (ID: {element.id})")
416+
console.print(f" [dim]Role: {element.role}, Clickable: {element.visual_cues.is_clickable}[/dim]")
417+
418+
# Click the element using the authorized action pattern
419+
# Post-verification will automatically check that URL contains "example-domains" after click
420+
await self._authorized_action(
421+
action="click",
422+
target=f"element#{element.id}",
423+
executor=lambda: self._click_element(element), # Returns coroutine
424+
)
425+
426+
async def _click_element(self, element):
427+
"""Click an element by its ID."""
428+
# Use Playwright's selector to click the element
429+
# The element.id is the unique identifier from the snapshot
430+
selector = f"[data-sentience-id='{element.id}']"
431+
432+
try:
433+
await self.browser.page.click(selector, timeout=5000)
434+
console.print(f" [dim]Clicked element with selector: {selector}[/dim]")
435+
except Exception as e:
436+
# Fallback: try clicking by XPath or other means
437+
console.print(f" [yellow]⚠[/yellow] Direct click failed, trying alternative: {e}")
438+
# Use bounding box to click by coordinates
439+
await self.browser.page.mouse.click(
440+
element.bbox.x + element.bbox.width / 2,
441+
element.bbox.y + element.bbox.height / 2,
442+
)
443+
console.print(f" [dim]Clicked at coordinates: ({element.bbox.x}, {element.bbox.y})[/dim]")
444+
318445
async def _get_page_summary(self) -> str:
319446
"""Get summary of current page state."""
320447
if not self.browser or not self.browser.page:
@@ -428,6 +555,18 @@ async def _cleanup(self):
428555
except Exception as e:
429556
logger.warning(f"Error closing browser: {e}")
430557

558+
# Close cloud tracer (blocking to ensure upload completes)
559+
if self.tracer:
560+
try:
561+
console.print("[dim]Uploading trace to Predicate Studio...[/dim]")
562+
self.tracer.close(blocking=True)
563+
console.print("[green]✓[/green] Trace uploaded")
564+
console.print(
565+
f" [dim]View in Studio: https://studio.predicatesystems.dev/runs/{self.run_id}[/dim]"
566+
)
567+
except Exception as e:
568+
logger.warning(f"Error closing tracer: {e}")
569+
431570

432571
async def main():
433572
"""Main entry point."""

0 commit comments

Comments
 (0)