YM-usr_template

Workspace Mapping

workspace-建图 is translated as workspace-mapping.

This workspace opens a local mapping window from main.py. The left side shows aircraft image frames, and the right side shows a stitched visual map with the current camera footprint and trajectory.

The default runtime image source is auto. It first reads the saved RenesasAgent camera setting from:

C:\Users\<you>\Documents\RenesasAgent\config.json

For the standard ESP32 camera stream this usually resolves to an MJPEG URL such as:

http://192.168.10.20:81/stream

Supported frame sources:

mjpeg    HTTP MJPEG stream
snapshot HTTP image polling
rtsp     RTSP/video URL through OpenCV
local    local camera index or video file through OpenCV
sdk      yueming_sdk UDP 4211 USER_RX image bytes

The SDK source is still available for experiments that send compressed image bytes through yueming_sdk.receive_data(). For large images, the sender should split a JPEG frame into chunks. This workspace accepts the optional RAIMG1 chunk header:

RAIMG1 + frame_id:u16 + chunk_index:u16 + total_chunks:u16 + jpeg_payload_chunk

Backend Selection

auto backend
  -> ORB-SLAM3 visual-inertial if configured
  -> RTAB-Map if installed
  -> DROID-SLAM / DPVO if configured
  -> ArUco metric localization if camera calibration exists
  -> OpenCV stitched-mosaic fallback only when nothing professional is available

The OpenCV fallback now builds a visual mosaic map from real frames and consumes RA/ESP inertial telemetry when available. It is still relative-only: no metric scale, no loop closure, and no global map optimization.

Inertial Telemetry

During mapping, the app polls RA/ESP telemetry:

UDP 4210 MFTEL1 -> roll/pitch/yaw, gx/gy/gz, ax/ay/az, altitude, vertical speed
HTTP /api/overview -> fallback telemetry source

The telemetry host is inferred from the video URL by default. Override it when needed:

python main.py --telemetry-host 192.168.10.20 --telemetry-port 4210 --telemetry-hz 20

Telemetry is logged to output/inertial_telemetry.csv unless disabled:

python main.py --no-telemetry

Run With Aircraft Image Link

Default connection using the saved RenesasAgent video source:

python D:\renesas_project\workspace-mapping\main.py --source-type auto

Manual MJPEG stream:

python D:\renesas_project\workspace-mapping\main.py --source-type mjpeg --source-url http://192.168.10.20:81/stream

Manual SDK USER_RX source:

python D:\renesas_project\workspace-mapping\main.py --source-type sdk --esp-host 192.168.4.1 --esp-port 4211 --local-port 0

Headless smoke style run against the SDK stream:

python D:\renesas_project\workspace-mapping\main.py --headless --frames 36 --source-type auto

Save a final map image:

python D:\renesas_project\workspace-mapping\main.py --headless --frames 60 --source-type auto --save-map D:\renesas_project\workspace-mapping\output\map.png

Red Ball Follower

Manual keyboard control, matching the RA-rensen upper-computer keys:

python manual_flight_keyboard.py

Keys:

Space      lock/unlock
W / S      up / down       -> vspeed +/-
A / D      yaw left/right  -> yaw -/+
Arrow Up   forward         -> pitch +
Arrow Down back            -> pitch -
Arrow Left left            -> roll -
Arrow Right right          -> roll +

Preview packet output without sending UDP:

python manual_flight_keyboard.py --dry-run

Keyboard-priority red-ball assist:

python manual_flight_keyboard.py --auto-track

Track any object instead of only a red ball:

python manual_flight_keyboard.py --auto-track --target-mode roi

In the manual-flight window, press T to draw a target box and C to clear it. If you start red_ball_follower.py directly, press S in the preview window to switch into ROI tracking and draw a box around the object.

For stronger arbitrary-object tracking, use the UETrack backend:

python manual_flight_keyboard.py --auto-track --tracker uetrack-auto

uetrack-auto first tries UETrack and falls back to the built-in OpenCV tracker if PyTorch or checkpoints are not installed yet. Force UETrack only with:

python manual_flight_keyboard.py --auto-track --tracker uetrack --uetrack-checkpoint D:\models\UETrack_ep0500.pth.tar

In this mode all MFUDP1 packets come from manual_flight_keyboard.py. Keyboard input has priority over vision: any WASD/arrow key press immediately interrupts the automatic red-ball assist for a short holdoff window. The vision assist only contributes roll/pitch when the red ball target is fresh; target lost/stale does not send disarm packets or its own zero-control stream.

Tune the automatic red-ball P gain with one combined knob:

python manual_flight_keyboard.py --auto-track --auto-gain 2.0

Keep the P gain high but limit the maximum automatic roll/pitch output:

python manual_flight_keyboard.py --auto-track --auto-gain 2.4 --auto-max-axis 0.35

red_ball_follower.py reads the same aircraft image sources as main.py, detects the largest solid red ball in the frame, maps the calibrated front/back/left/right image position to bounded roll and pitch controls, and can send RA-rensen MFUDP1 packets over UDP port 4210.

The default mode is --mode ground: manually fly the aircraft above the target area first, then let the script begin automatic roll/pitch tracking after the red ball is detected. The calibration defaults are written from the collected samples in output/red_ball_calibration/redo_20260624_023237.

Control sending defaults to --control-backend auto, which uses the same D:\KAI\Desktop\Project\RA-rensen\renesas-agent\upmachine_demo_pre\src\yueming_sdk.flight path as the upper-computer Code page when that SDK is present. This keeps the MFUDP1 packet format, host fallback, and flight-control API aligned with the upper-computer link. --control-backend local is only a fallback/debug path.

Preview only, with the saved RenesasAgent camera setting:

python red_ball_follower.py --source-type auto

Preview any manually selected object:

python red_ball_follower.py --source-type auto --target-mode roi

Try the stronger UETrack backend while keeping automatic fallback:

python red_ball_follower.py --source-type auto --target-mode roi --tracker uetrack-auto

Preview an ESP32 MJPEG stream:

python red_ball_follower.py --source-type mjpeg --source-url http://192.168.10.20:81/stream

If madflight.local is unstable on your network, prefer the direct IP:

E:\AI\conda-envs\uetrack-py310\python.exe manual_flight_keyboard.py --auto-track --tracker uetrack --source-type mjpeg --source-url http://192.168.10.20:81/stream --control-host 192.168.10.20

Preview without a window, useful for tests or logs:

python red_ball_follower.py --source-type auto --headless --frames 120

Standalone red_ball_follower.py --send is disabled by default because it can compete with keyboard control. Use manual_flight_keyboard.py --auto-track for flight. Standalone send is only for bench testing with propellers removed.

Send real UDP control packets from the standalone tracker only after a manual safety check:

python red_ball_follower.py `
  --source-type mjpeg `
  --source-url http://192.168.10.20:81/stream `
  --control-host 192.168.10.20 `
  --control-backend auto `
  --send `
  --i-know-standalone-send-is-dangerous

By default --send still sends armed=false. Add --armed only when the aircraft is physically safe, a human is ready to stop it, and the output in preview mode already looks correct. Target lost or low confidence forces all motion axes back to zero.

Tuning knobs:

python red_ball_follower.py `
  --source-type auto `
  --min-area 180 `
  --max-axis 0.25 `
  --yaw-gain 0.55 `
  --vertical-gain 0.45 `
  --target-radius-ratio 0.10

Control mapping:

ball at calibrated right -> positive roll
ball at calibrated left  -> negative roll
ball at calibrated front -> positive pitch
ball at calibrated back  -> negative pitch
target lost             -> all axes zero

The old front-camera centering behavior is still available with --mode camera.

UETrack Setup

This workspace now contains the official UETrack source in external/UETrack. To actually run the UETrack backend you still need its runtime dependencies and checkpoint:

pip install torch torchvision timm yacs
pip install git+https://github.com/openai/CLIP.git

This machine is now configured with a ready-to-use CPU UETrack environment on:

E:\AI\conda-envs\uetrack-py310

and the downloaded model files on:

E:\AI\models\UETrack\hf\uetrack

Then download a UETrack checkpoint and pass it explicitly:

python red_ball_follower.py `
  --source-type auto `
  --target-mode roi `
  --tracker uetrack `
  --uetrack-config uetrack_tiny `
  --uetrack-checkpoint D:\models\UETrack_ep0500.pth.tar

If you use --tracker uetrack-auto, the program will fall back to the existing OpenCV ROI tracker whenever UETrack cannot be initialized.

On this machine you can now directly use:

E:\AI\conda-envs\uetrack-py310\python.exe manual_flight_keyboard.py --auto-track --tracker uetrack

Precision Mode

For real precision, do not rely on the fallback. Use one of these:

• --backend orbslam3 with ORB-SLAM3 executable, vocabulary, and camera settings.
• --backend rtabmap with RTAB-Map installed.
• --backend droidslam or --backend dpvo with GPU-ready deep SLAM environments.
• --backend aruco_metric with a calibrated camera and known-size ArUco markers visible in the aircraft image stream.

Example ArUco metric localization from the aircraft image stream:

python D:\renesas_project\workspace-mapping\main.py `
  --backend aruco_metric `
  --camera-config D:\renesas_project\workspace-mapping\config\camera.yaml `
  --source-type auto

Example ORB-SLAM3 visual-inertial configuration:

python D:\renesas_project\workspace-mapping\main.py `
  --backend orbslam3 `
  --orbslam3-bin D:\slam\ORB_SLAM3\Examples\Monocular-Inertial\mono_inertial.exe `
  --orbslam3-vocab D:\slam\ORB_SLAM3\Vocabulary\ORBvoc.txt `
  --orbslam3-imu-settings D:\renesas_project\workspace-mapping\config\orbslam3_visual_inertial.yaml `
  --source-type auto `
  --telemetry-host 192.168.10.20

If a requested professional backend is missing, the program fails with a clear diagnostic. It will not silently downgrade a precision request to optical flow.

Calibration

Generate a printable ChArUco board:

python tools\generate_charuco_board.py --output output\charuco_board.png

Calibrate the same camera/lens used by the aircraft image stream:

python tools\calibrate_camera.py --source camera --source-value 0 --output config\camera.yaml --samples 35

Generate an ArUco marker:

python tools\generate_aruco_marker.py --id 0 --output output\aruco_marker_0.png

Measure the printed marker side length and set marker_length_m in config/camera.yaml.

Files

workspace-mapping/
  main.py
  mapping_app/
    backends/
      base.py
      factory.py
      aruco_metric.py
      external.py
    config.py
    sources.py        # MJPEG/snapshot/RTSP/local/SDK frame sources
    ui.py
    visual_odometry.py
  config/
    camera.example.yaml
  docs/
    PROFESSIONAL_SLAM_BACKENDS.md
  tools/
    generate_charuco_board.py
    generate_aruco_marker.py
    calibrate_camera.py
  tests/
    smoke_mapping.py
    smoke_aruco_metric.py

The upper-computer Code page only needs to run main.py. All other files are normal Python modules imported by main.py.

Accuracy Notes

• A monocular aircraft camera alone cannot provide reliable absolute scale.
• For precise localization, use calibrated stereo/RGB-D/IMU, or calibrated fiducials with known physical size.
• Fast rotation, blank floors, motion blur, rolling shutter, and lighting jumps hurt visual localization.
• Validate precision against measured ground truth. A nice-looking trajectory is not proof of accuracy.