Merge pull request #248 from ryanbreen/feat/vmware-support

feat: VMware Fusion ARM64 support — full boot, XHCI USB keyboard + mouse, cursor

Author: ryanbreen

VMWARE_HANDOFF.md

@@ -0,0 +1,124 @@
+# VMware Fusion Support — Handoff Document
+
+**Branch:** `feat/vmware-support` (branched from `feat/full-gpu-compositing` at `c3744ea`)
+**Worktree:** `/Users/wrb/fun/code/breenix-vmware`
+**Date:** 2026-03-06
+
+## What's Done
+
+### 1. `run.sh --vmware` flag (COMPLETE)
+
+Full VMware Fusion integration added to `run.sh`, mirroring the `--parallels` pattern:
+
+- Builds UEFI loader + ARM64 kernel + userspace + ext2 disk (same pipeline as Parallels)
+- Creates FAT32 ESP image with GPT ESP patch
+- Converts raw images to VMDK via `qemu-img convert -f raw -O vmdk`
+- Generates `.vmwarevm` bundle with unique timestamped name
+- Cleans up old `breenix-*` VMs automatically
+- Launches via `vmrun start`, tails serial log
+
+Usage:
+```bash
+./run.sh --vmware              # Full build + boot
+./run.sh --vmware --no-build   # Reuse last build
+./run.sh --vmware --clean      # Clean rebuild
+```
+
+### 2. VMX Generator (COMPLETE)
+
+`scripts/vmware/generate-vmx.sh` — generates a `.vmx` config file with:
+- ARM64 EFI guest (`guestOS = "arm-other-64"`, `firmware = "efi"`)
+- NVMe boot disk (boot.vmdk with FAT32 ESP)
+- SATA ext2 data disk (optional)
+- Serial port output to `/tmp/breenix-vmware-serial.log`
+- NAT networking (e1000e)
+- XHCI USB
+- 4 vCPUs, 2GB RAM, 256MB VRAM
+
+### 3. First Boot Test (DONE — partial success)
+
+The kernel DOES boot on VMware Fusion. Evidence from `vmware.log`:
+- `Guest: Firmware has transitioned to runtime` — UEFI loader executed
+- vcpu-0 active with valid state: `PC=fbce8774`, `VBAR_EL1=ff3dd000`, `TTBR0_EL1=fffdf000`
+- MMU enabled (`SCTLR_EL1=3050198d`), page tables set up
+- Screen resized: `Screen 1 Defined: xywh(0, 0, 2048, 1536)`
+
+## What's Broken
+
+### Serial Output — GARBLED (not GPU-related)
+
+The serial log (`/tmp/breenix-vmware-serial.log`) contains non-ASCII binary garbage (bytes in 0x00-0x1b range). The kernel is running but we can't see any output.
+
+**Root cause:** The UEFI loader discovers the UART via ACPI SPCR table. When SPCR isn't found or doesn't match, it falls back to the Parallels PL011 address `0x0211_0000`. VMware Fusion almost certainly has its UART at a different base address.
+
+**Key code path:**
+1. `parallels-loader/src/acpi_discovery.rs:187-219` — SPCR parsing, falls back to `0x0211_0000`
+2. `kernel/src/platform_config.rs:22` — QEMU default is `0x0900_0000`
+3. `kernel/src/serial_aarch64.rs` — PL011 driver (may need 16550 for VMware)
+
+## What Needs To Happen Next
+
+### Priority 1: Fix Serial Output
+
+This is the blocker — without serial we're flying blind.
+
+**Option A: Debug from UEFI loader side**
+- The UEFI loader has access to UEFI SimpleTextOutput (screen console) before handing off
+- Add SPCR table dump logging in `parallels-loader/src/acpi_discovery.rs` using UEFI output services
+- This will reveal what ACPI tables VMware provides and what UART address is in SPCR
+
+**Option B: Dump ACPI tables from VMware**
+- Boot a Linux ISO on the same VMware VM config
+- Run `cat /sys/firmware/acpi/tables/SPCR | xxd` to see the UART address
+- Or `dmesg | grep -i uart` to see what Linux discovers
+
+**Option C: Try known VMware UART addresses**
+- VMware ARM64 may use PL011 at `0x0900_0000` (same as QEMU virt)
+- Or it might use 16550-compatible UART at a different address
+- Could try hardcoding common addresses as a quick test
+
+### Priority 2: Verify UART Type
+
+VMware might not use PL011 at all on ARM64. It could use:
+- **16550 UART** (like x86) — different register layout, different init sequence
+- **PL011** at a different address — just need the right base
+- Check SPCR `interface_type` field: 0x03 = PL011, 0x12 = 16550
+
+### Priority 3: Platform Abstraction
+
+Once serial works, consider:
+- Adding a `VMware` variant to platform detection (currently just QEMU vs Parallels)
+- VMware's virtio/GPU/network devices may differ from Parallels
+- The existing Breenix VM at `~/Virtual Machines.localized/Breenix.vmwarevm/Breenix.vmx` has a known-working VMware config for reference
+
+## File Inventory
+
+### New files
+- `scripts/vmware/generate-vmx.sh` — VMX config generator
+- `VMWARE_HANDOFF.md` — this document
+
+### Modified files
+- `run.sh` — added `--vmware` flag, VMWARE block (lines ~410-610)
+
+### Build artifacts
+- `target/vmware/boot.vmdk` — EFI boot disk
+- `target/vmware/ext2-data.vmdk` — ext2 data disk
+- VM bundles land in `~/Virtual Machines.localized/breenix-*.vmwarevm/`
+
+## VMware Fusion Tooling Reference
+
+| Tool | Path | Purpose |
+|------|------|---------|
+| vmrun | `/Applications/VMware Fusion.app/Contents/Public/vmrun` | VM lifecycle (start/stop) |
+| vmware-vdiskmanager | `/Applications/VMware Fusion.app/Contents/Library/vmware-vdiskmanager` | VMDK creation/conversion |
+| vmcli | `/Applications/VMware Fusion.app/Contents/Public/vmcli` | VM configuration |
+| EFI ROM | `.../Library/roms/arm64/EFIAARCH64.ROM` | ARM64 EFI firmware |
+
+Disk conversion uses `qemu-img` (already installed at `/opt/homebrew/bin/qemu-img`) rather than vmware-vdiskmanager, since qemu-img handles raw-to-vmdk directly.
+
+## Existing VMware VM (Manual Setup)
+
+There's an existing manually-created Breenix VM at:
+`~/Virtual Machines.localized/Breenix.vmwarevm/Breenix.vmx`
+
+This was set up before the automated `--vmware` flow. It has the ext2 image attached as `sata0:1` (cdrom-image pointing at `target/ext2-aarch64.img`). The `.vmx` from that VM was used as reference for `generate-vmx.sh`. The VM is currently suspended with a checkpoint.

kernel/src/arch_impl/aarch64/boot.S

@@ -223,7 +223,7 @@ curr_el_spx_irq:
     b irq_handler
 .balign 0x80
 curr_el_spx_fiq:
-    b unhandled_exception
+    b irq_handler
 .balign 0x80
 curr_el_spx_serror:
     b unhandled_exception
@@ -237,7 +237,7 @@ lower_el_aarch64_irq:
     b irq_handler
 .balign 0x80
 lower_el_aarch64_fiq:
-    b unhandled_exception
+    b irq_handler
 .balign 0x80
 lower_el_aarch64_serror:
     b unhandled_exception

kernel/src/arch_impl/aarch64/cpu.rs

@@ -31,28 +31,32 @@ const DAIF_ALL_IMM: u32 = 0xF;  // D, A, I, F
 pub struct Aarch64Cpu;
 
 impl CpuOps for Aarch64Cpu {
-    /// Enable IRQ interrupts by clearing the I bit in DAIF
+    /// Enable IRQ and FIQ interrupts by clearing the I and F bits in DAIF.
+    ///
+    /// Both IRQ and FIQ must be unmasked because on VMware Fusion, all GIC
+    /// interrupts are Group 0 (FIQ) due to GICR_IGROUPR0 being RAZ/WI.
     #[inline]
     unsafe fn enable_interrupts() {
-        // daifclr with #2 clears the I bit (enables IRQs)
-        core::arch::asm!("msr daifclr, #2", options(nomem, nostack));
+        // daifclr with #3 clears bits I and F (enables both IRQs and FIQs)
+        core::arch::asm!("msr daifclr, #3", options(nomem, nostack));
     }
 
-    /// Disable IRQ interrupts by setting the I bit in DAIF
+    /// Disable IRQ and FIQ interrupts by setting the I and F bits in DAIF
     #[inline]
     unsafe fn disable_interrupts() {
-        // daifset with #2 sets the I bit (disables IRQs)
-        core::arch::asm!("msr daifset, #2", options(nomem, nostack));
+        // daifset with #3 sets bits I and F (disables both IRQs and FIQs)
+        core::arch::asm!("msr daifset, #3", options(nomem, nostack));
     }
 
-    /// Check if IRQ interrupts are enabled (I bit is clear)
+    /// Check if interrupts are enabled (both I and F bits are clear)
     #[inline]
     fn interrupts_enabled() -> bool {
         let daif: u64;
         unsafe {
             core::arch::asm!("mrs {}, daif", out(reg) daif, options(nomem, nostack));
         }
-        // IRQs are enabled when the I bit (bit 7) is clear
+        // Interrupts are enabled when the I bit (bit 7) is clear
+        // (FIQ is also unmasked but we check I as the primary indicator)
         (daif & DAIF_IRQ_BIT) == 0
     }
 
@@ -76,10 +80,10 @@ impl CpuOps for Aarch64Cpu {
     #[inline]
     fn halt_with_interrupts() {
         unsafe {
-            // Enable IRQs and immediately wait
+            // Enable IRQs/FIQs and immediately wait
             // Any pending interrupt will be taken before WFI completes
             core::arch::asm!(
-                "msr daifclr, #2",  // Enable IRQs
+                "msr daifclr, #3",  // Enable IRQs and FIQs
                 "wfi",              // Wait for interrupt
                 options(nomem, nostack)
             );
@@ -101,22 +105,18 @@ impl CpuOps for Aarch64Cpu {
             core::arch::asm!("mrs {}, daif", out(reg) daif, options(nomem, nostack));
         }
 
-        // Disable IRQs
+        // Disable IRQs and FIQs
         unsafe {
-            core::arch::asm!("msr daifset, #2", options(nomem, nostack));
+            core::arch::asm!("msr daifset, #3", options(nomem, nostack));
         }
 
         // Execute the closure
         let result = f();
 
-        // Restore previous DAIF state (only restore IRQ bit to avoid affecting other flags)
-        if (daif & DAIF_IRQ_BIT) == 0 {
-            // IRQs were enabled before, re-enable them
-            unsafe {
-                core::arch::asm!("msr daifclr, #2", options(nomem, nostack));
-            }
+        // Restore previous DAIF state exactly
+        unsafe {
+            core::arch::asm!("msr daif, {}", in(reg) daif, options(nomem, nostack));
         }
-        // If IRQs were disabled, leave them disabled (don't change anything)
 
         result
     }

kernel/src/arch_impl/aarch64/exception.rs

@@ -974,9 +974,9 @@ pub extern "C" fn handle_irq() {
     if let Some(irq_id) = gic::acknowledge_irq() {
         // Handle the interrupt based on ID
         match irq_id {
-            // Virtual timer interrupt (PPI 27)
+            // Timer interrupt — virtual (PPI 27) or physical (PPI 30)
             // This is the scheduling timer - calls into scheduler
-            crate::arch_impl::aarch64::timer_interrupt::TIMER_IRQ => {
+            irq if irq == crate::arch_impl::aarch64::timer_interrupt::timer_irq() => {
                 // Call the timer interrupt handler which handles:
                 // - Re-arming the timer
                 // - Updating global time
@@ -999,7 +999,24 @@ pub extern "C" fn handle_irq() {
             }
 
             // PPIs (16-31) - Private peripheral interrupts (excluding timer)
-            16..=31 => {}
+            // On VMware, we enable both virtual (27) and physical (30) timers
+            // to discover which fires. Handle either as a timer interrupt.
+            irq @ 16..=31 => {
+                if irq == crate::arch_impl::aarch64::timer_interrupt::VIRT_TIMER_IRQ
+                    || irq == crate::arch_impl::aarch64::timer_interrupt::PHYS_TIMER_IRQ
+                {
+                    crate::arch_impl::aarch64::timer_interrupt::timer_interrupt_handler();
+                }
+                // Diagnostic: emit raw serial for any unexpected PPI
+                if irq != crate::arch_impl::aarch64::timer_interrupt::timer_irq()
+                    && irq != crate::arch_impl::aarch64::timer_interrupt::VIRT_TIMER_IRQ
+                    && irq != crate::arch_impl::aarch64::timer_interrupt::PHYS_TIMER_IRQ
+                {
+                    crate::serial_aarch64::raw_serial_char(b'P');
+                    crate::serial_aarch64::raw_serial_char(b'0' + (irq / 10) as u8);
+                    crate::serial_aarch64::raw_serial_char(b'0' + (irq % 10) as u8);
+                }
+            }
 
             // SPIs (32-1019) - Shared peripheral interrupts
             // Note: No logging here - interrupt handlers must be < 1000 cycles