Add support for basic camera streams [SYNTH-10]

[PepperLola]

Task

SYNTH-10

Added the ability to add cameras to the robot and both preview their streams from within Fission and stream them to the robot code via WebSocket. Since HALSim doesn't support cameras, this involves spinning up another WebSocket server on the robot side and streaming the video feed to it from Fission.

Verification

Spawn a robot in Fission and set the brain to the WPILib one. Configure cameras on the robot (within robot configuration - place camera, edit settings). Set up code sim by following the document added in #1360 and keep track of the mean brightness (calculated on the robot code side) in the simulator.

---

Before merging, ensure the following criteria are met:

• All acceptance criteria outlined in the ticket are met.
• Necessary test cases have been added and updated.
• A feature toggle or safe disable path has been added (if applicable).
• User-facing polish:
  • Ask: "Is this ready-looking?"
• Cross-linking between Jira and GitHub:
  • PR links to the relevant Jira issue.
  • Jira ticket has a comment referencing this PR.

[Copilot]

Pull request overview

Adds end-to-end support for simulated “USB cameras” in Synthesis: camera configuration is published via HALSim (SimDevice), while rendered frames are streamed over a side-channel WebSocket to the robot process so existing WPILib/OpenCV vision code can consume frames.

Changes:

• Introduces a simulated UsbCamera/CvSink implementation in SyntheSimJava backed by a local WebSocket frame receiver and HALSim-published camera config.
• Adds Fission UI + scene rendering support to configure cameras on a robot, preview them in-app, and stream JPEG frames to the robot-side WebSocket server.
• Extends preference types/tests and WPILib sim plumbing with a new SimType.CAMERA and camera config field constants.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
simulation/SyntheSimJava/src/main/java/com/autodesk/synthesis/cscore/UsbCamera.java	Adds a drop-in UsbCamera wrapper that sources frames from Synthesis instead of hardware.
simulation/SyntheSimJava/src/main/java/com/autodesk/synthesis/cscore/CvSink.java	Adds a CvSink wrapper that returns decoded frames from the simulated camera.
simulation/SyntheSimJava/src/main/java/com/autodesk/synthesis/cscore/CameraFrameServer.java	Implements localhost WebSocket server to receive <device>\n<base64-jpeg> frame messages.
simulation/SyntheSimJava/src/main/java/com/autodesk/synthesis/cscore/Camera.java	Publishes camera config via SimDevice and decodes latest JPEG bytes into an OpenCV Mat.
simulation/SyntheSimJava/gradlew.bat	Adds SPDX license tag comment.
simulation/SyntheSimJava/gradlew	Adds SPDX license tag comment + minor wrapper script adjustments.
simulation/SyntheSimJava/gradle/wrapper/gradle-wrapper.properties	Updates Gradle wrapper distribution URL.
simulation/SyntheSimJava/build.gradle	Adds cscore/OpenCV + Java-WebSocket dependencies for camera streaming.
simulation/samples/JavaAutoSample/src/main/java/frc/robot/Robot.java	Demonstrates grabbing frames, computing mean brightness, and republishing via CameraServer.
fission/src/ui/panels/simulation/CameraPreviewPanel.tsx	Adds a panel to preview configured robot cameras by copying their frame canvases.
fission/src/ui/panels/configuring/assembly-config/interfaces/ConfigureCameraInterface.tsx	Adds configuration UI for camera name, parent node transform, FOV, resolution, and FPS.
fission/src/ui/panels/configuring/assembly-config/ConfigurePanel.tsx	Wires the new “USB Cameras” config mode into the Configure panel.
fission/src/ui/panels/configuring/assembly-config/ConfigTypes.ts	Adds ConfigMode.CAMERA.
fission/src/test/PreferencesSystem.test.ts	Updates expected robot preference defaults to include cameras: [].
fission/src/systems/simulation/wpilib_brain/WPILibTypes.ts	Adds SimType.CAMERA + camera config field keys.
fission/src/systems/simulation/wpilib_brain/WPILibState.ts	Extends supplier/receiver type maps for the new camera sim type.
fission/src/systems/simulation/wpilib_brain/sim/SimCamera.ts	Adds helper getters for camera width/height/fps and presence via sim map.
fission/src/systems/simulation/wpilib_brain/sim/SimAccel.ts	Fixes Jolt memory ownership issues when reading rotation/velocity (avoid destroying borrowed refs).
fission/src/systems/simulation/wpilib_brain/CameraFrameSocket.ts	Implements outgoing WS client to stream frames to the robot-side server with backpressure dropping.
fission/src/systems/preferences/PreferenceTypes.ts	Adds CameraPreferences and integrates it into RobotPreferences + defaults.
fission/src/mirabuf/RobotCameraSceneObject.ts	Renders camera views to a render target, previews to a canvas, and streams JPEG frames to robot.
fission/src/mirabuf/MirabufSceneObject.ts	Creates/destroys camera scene objects and exposes camera preview panel entry when cameras exist.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 24 changed files in this pull request and generated 6 comments.

[AlexD717]

Overall functionality seems pretty good. I am able to add a camera and view its stream.

However, when I try and edit the camera FPS, width, and height, it doesn't seem to update. The field of view does update.

In addition, the frame received value alternates every frame from false to true (this is very minor so don't bother fixing if its hard).

[PepperLola]

However, when I try and edit the camera FPS, width, and height, it doesn't seem to update. The field of view does update.

Do you mean update them in Fission? or through Shuffleboard? I'll make sure that works when updating in Fission, but I don't plan to support that in Shuffleboard (i.e., the FPS and Resolution inputs in the camera widget).

In addition, the frame received value alternates every frame from false to true (this is very minor so don't bother fixing if its hard).

This is all on the side of the robot code and is intended behavior, but I could still change it if it's misleading from a code sample point of view. The robot won't receive frames at the same rate as its periodic methods are called, and the frame received value is true only for a tick where there is a new frame. If the robot is updating at 50tps but the camera is streaming at 30fps, even under nominal network conditions they wouldn't line up and you'd end up with some ticks where there is "no frame received" even though everything's working properly. If it makes more sense to remove this from the sample or replace it with a different metric then I can do that.

[AlexD717]

However, when I try and edit the camera FPS, width, and height, it doesn't seem to update. The field of view does update.

Do you mean update them in Fission? or through Shuffleboard? I'll make sure that works when updating in Fission, but I don't plan to support that in Shuffleboard (i.e., the FPS and Resolution inputs in the camera widget).

I meant in fission.

In addition, the frame received value alternates every frame from false to true (this is very minor so don't bother fixing if its hard).

This is all on the side of the robot code and is intended behavior, but I could still change it if it's misleading from a code sample point of view. The robot won't receive frames at the same rate as its periodic methods are called, and the frame received value is true only for a tick where there is a new frame. If the robot is updating at 50tps but the camera is streaming at 30fps, even under nominal network conditions they wouldn't line up and you'd end up with some ticks where there is "no frame received" even though everything's working properly. If it makes more sense to remove this from the sample or replace it with a different metric then I can do that.

I wouldn't bother, this was just a minor thing I noticed.

[rutmanz]

Doesn't seem to support multiple camera streams on the dashboards

[Dhruv-0-Arora]

multiple camera views in shuffleboard.
Only one camera currently shows in shuffleboard despite when you have multiple spawned