Real-time remote assistance platform using Meta Ray-Ban smart glasses
A complete system that enables remote experts to see through Meta Ray-Ban smart glasses, provide voice guidance, and draw real-time annotations that appear in the wearer's field of view.
Remote Pair-Eyes transforms Meta Ray-Ban glasses into a powerful remote assistance tool. Field technicians, workers, or anyone wearing the glasses can stream their first-person view to remote experts who can guide them with voice chat and visual annotations.
- Field Service: Technicians get real-time expert guidance for complex repairs
- Medical: Remote specialists assist with procedures
- Training: Instructors guide trainees hands-free
- Accessibility: Visual assistance for those who need it
- Industrial: Remote inspection and quality control
- Live camera feed from Meta Ray-Ban glasses
- Low-latency WebSocket-based streaming
- Automatic quality optimization
- Frame rate up to 30 FPS
- Full-duplex WebRTC audio
- Push-to-talk or always-on modes
- Mute/unmute controls
- Works from both phone and dashboard
- Pointer: Quick attention indicator with auto-fade
- Arrow: Directional guidance that persists
- Circle: Highlight areas of interest
- Rectangle: Frame specific components
- Freehand Drawing: Complex instructions
- Text Labels: Add contextual notes
Annotations appear on the phone overlay and sync in real-time across all participants.
- Access code system for secure connections
- Multiple participants per room
- Role-based permissions (Wearer, Expert, Viewer)
- Easy sharing via access codes
- Text-to-speech instructions on the phone
- Audio cues when experts join/leave
- Voice announcements for room events
- Floating action button for easy access
- Photo capture
- Audio recording
- Mute toggle
- Call controls
- No scrolling needed - overlay design
┌─────────────────┐ WebSocket ┌─────────────────┐
│ Meta Glasses │◄──────────────────►│ Backend │
│ (via Phone) │ │ (Node.js) │
└─────────────────┘ └────────┬────────┘
│ │
│ Camera Feed │ WebSocket
│ + Annotations │
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Android App │ │ Dashboard │
│ (Kotlin) │ │ (React) │
└─────────────────┘ └─────────────────┘
▲ │
│ WebRTC Audio │
└───────────────────────────────────────┘
metasdk/
├── metatest/ # Android companion app (Kotlin)
│ ├── app/
│ │ └── src/main/java/com/example/metatest/
│ │ ├── MainActivity.kt # Main app with camera, WebRTC, annotations
│ │ └── crypto/
│ │ └── EncryptionManager.kt
│ └── CLAUDE.md # Development notes
│
└── meta-glasses-companion/ # Backend + Dashboard (TypeScript)
├── apps/
│ ├── backend/ # WebSocket server (port 8000)
│ │ └── src/
│ │ ├── server.ts
│ │ ├── websocket/WebSocketServer.ts
│ │ └── rooms/RoomService.ts
│ │
│ └── dashboard/ # React web app (port 5173)
│ └── src/
│ ├── components/
│ │ ├── AnnotationCanvas.tsx
│ │ ├── VideoFeed.tsx
│ │ └── VoiceChat.tsx
│ └── stores/
│ ├── sessionStore.ts
│ └── encryptionStore.ts
│
└── packages/
└── shared/ # Shared types & utilities
└── src/
└── types/
├── messages.ts
└── room.ts
- Node.js 18+
- pnpm
- Android Studio (for mobile app)
- Meta Ray-Ban glasses paired with phone
- Meta View app installed
cd meta-glasses-companion
# Install dependencies
pnpm install
# Start development servers
pnpm dev
# Backend runs on http://localhost:8000
# Dashboard runs on http://localhost:5173- Open
metatest/in Android Studio - Update backend IP in
MainActivity.kt:private val backendHost = "YOUR_IP_ADDRESS"
- Build and install on your phone
- Ensure Meta glasses are connected via Meta View app
- Start the servers: Run
pnpm devin meta-glasses-companion - Open dashboard: Navigate to
http://localhost:5173 - Launch phone app: Open Remote Pair-Eyes on your phone
- Connect glasses: Ensure Meta View shows glasses as connected
- Start streaming: Tap "Start Streaming" on the phone
- Join from dashboard: Enter the access code shown on phone
- Collaborate: Use voice chat and annotations to guide the wearer
- Kotlin + Jetpack Compose
- Meta SDK for glasses camera access
- WebRTC for voice communication
- Ktor for WebSocket client
- Material3 UI components
- Node.js + TypeScript
- WebSocket (ws library)
- Express for HTTP endpoints
- React 18 + TypeScript
- Zustand for state management
- Tailwind CSS for styling
- Vite for bundling
- TypeScript types
- pnpm workspaces + Turborepo
PORT=8000private val backendHost = "192.168.x.x" // Your server IP
private val backendPort = 8000When developing in WSL, use PowerShell for adb commands:
powershell.exe -Command "adb devices"
powershell.exe -Command "adb install app.apk"
powershell.exe -Command "adb logcat -s RemotePairEyes:*"Glasses show as unavailable
- Ensure Meta View app is open and glasses are connected
- Try putting glasses in case for 30 seconds to reset
- Check for thermal throttling (let glasses cool down)
Stream stuck in STARTING
- Usually indicates thermal throttling
- Let glasses rest for a few minutes
- Check Meta View for error messages
WebSocket connection fails
- Verify backend is running on correct port
- Check firewall allows port 8000
- Ensure phone and server are on same network
MIT License - See LICENSE file for details
Contributions welcome! Please read CONTRIBUTING.md for guidelines.
Built with Meta SDK, WebRTC, and lots of coffee.