Merge pull request #120 from jumpstarter-dev/pytest-with-imagehash

soc testing example usign pytest and imagehash

Author: mangelajo

examples/soc-pytest/README.md

@@ -0,0 +1,176 @@
+# Jumpstarter SOC testing with pytest example
+
+This example aims to demonstrate Jumpstarter in an a simple
+SOC testing scenario using pytest.
+
+The following drivers will be utilized:
+- DUTLink: for power, storage and console control of the target
+- UStreamer: with an hdmi capture card plus a webcam for video snapshits
+
+This example requires the following hardware:
+- 1x Raspberry Pi 4
+- 1x dutlink  (DUTLink could be replaced by a composite set of power, storage mux and serial port interface)
+- 1x HDMI Capture card
+- 1x Webcam
+
+# Running the example (distributed env)
+
+1) Setup an environment with the required hardware, and customize the exporter.yaml
+2) Setup the exporter to be run from a container (TODO: link)
+3) Label the exporter in k8s with the `board=rpi4` label
+4) Prepare the images by running `make` in the `image` directory
+5) Run the tests in this directory by running:
+```bash
+$ cd jumpstarter_example_soc_pytest
+$ uv run pytest -s
+================================================================== test session starts ===================================================================
+platform linux -- Python 3.12.3, pytest-8.3.3, pluggy-1.5.0
+rootdir: /home/majopela/jumpstarter/examples/soc-pytest
+configfile: pyproject.toml
+plugins: anyio-4.6.2.post1, cov-5.0.0
+collected 6 items
+
+test_on_rpi4.py::TestResource::test_setup_device
+--------------------------------------------------------------------- live log setup ---------------------------------------------------------------------
+INFO     jumpstarter.client.lease:lease.py:35 Leasing Exporter matching labels {'board': 'rpi4'} for seconds: 1800
+
+INFO     jumpstarter.client.lease:lease.py:42 Lease c33b74ff-ad92-42a6-aa88-2c8a944a297c created
+INFO     jumpstarter.client.lease:lease.py:46 Polling Lease c33b74ff-ad92-42a6-aa88-2c8a944a297c
+INFO     jumpstarter.client.lease:lease.py:51 Lease c33b74ff-ad92-42a6-aa88-2c8a944a297c acquired
+INFO     jumpstarter.client.lease:lease.py:73 Connecting to Lease with name c33b74ff-ad92-42a6-aa88-2c8a944a297c
+--------------------------------------------------------------------- live log call ----------------------------------------------------------------------
+INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:51 Setting up device
+read: 2.45GB [00:49, 52.8MB/s]
+INFO     jumpstarter.testing.utils:utils.py:15 Waiting for login prompt
+
+RPi: BOOTLOADER release VERSION:817717 DATE: 2023/01/11 TIME: 17:40:52
+BOOTMODE: 0x06 partition 1 build-ts BUILD_TIMESTAMP=1673458852 serial c3656a7d boardrev d03114 stc 608563
+..
+Starting start4.elf @ 0xfeb00200 partition 1
++
+[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083]
+...
+...
+
+Raspbian GNU/Linux 12 rpitest ttyS0
+
+rpitest login: root
+Password:
+Linux rpitest 6.6.31+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.31-1+rpt1 (2024-05-29) aarch64
+
+The programs included with the Debian GNU/Linux system are free software;
+the exact distribution terms for each program are described in the
+individual files in /usr/share/doc/*/copyright.
+
+Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
+permitted by applicable law.
+root@rpitest:~#INFO     jumpstarter.testing.utils:utils.py:21 Logged in
+INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:112 Attempting a soft power off
+ poweroff
+root@rpitest:~#          Stopping session-1.scope - Session 1 of User root...
+...
+[   28.964752] reboot: Power down
+PASSED
+test_on_rpi4.py::TestResource::test_tpm2_device
+--------------------------------------------------------------------- live log setup ---------------------------------------------------------------------
+INFO     jumpstarter.testing.utils:utils.py:15 Waiting for login prompt
+INFO     jumpstarter.testing.utils:utils.py:21 Logged in
+--------------------------------------------------------------------- live log call ----------------------------------------------------------------------
+INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:78 Running command: apt-get install -y tpm2-tools
+apt-get install -y tpm2-tools
+...
+root@rpitest:~# INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:78 Running command: tpm2_createprimary -C e -c primary.ctx
+tpm2_createprimary -C e -c primary.ctx
+name-alg:
+  value: sha256
+  raw: 0xb
+attributes:
+  value: fixedtpm|fixedparent|sensitivedataorigin|userwithauth|restricted|decrypt
+  raw: 0x30072
+type:
+  value: rsa
+  raw: 0x1
+exponent: 65537
+bits: 2048
+scheme:
+  value: null
+  raw: 0x10
+scheme-halg:
+  value: (null)
+  raw: 0x0
+sym-alg:
+  value: aes
+  raw: 0x6
+sym-mode:
+  value: cfb
+  raw: 0x43
+sym-keybits: 128
+rsa: efe8d8387679d50d7cea501f4302834eebd4c4b3ec7f7b6a40128c63f3e9fb6e9203429dba4e1221d4d40039ff757dc3cbec638c79e11fe5cb4cc159a5e15a3d785b179f3081ada24f6370bad9b81ad2ddcba2e137bb62a454069d37da7cd1e3a06cb7fe03fc8386b055746b5396ee3b44aa1e40dae4e6257c763a53f7eb60a29df18ee14bce38d376434d89e9c95a79d1563833a48db8016c130f6246f24e023b8874e6f2f8bb1fbfe8ad9a1a0ef71b7fc0ed412056a40a225b6f352ea32aa9564c56bef09df7107b871db136aa530ae479b0b09256373479716416bc18fc7544df8c5de99383c37193f5e016bca7ab39231a69c6d4255d93aed66527bb261d
+root@rpitest:~# INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:78 Running command: tpm2_create -G rsa -u key.pub -r key.priv -C primary.ctx
+tpm2_create -G rsa -u key.pub -r key.priv -C primary.ctx
+name-alg:
+  value: sha256
+  raw: 0xb
+attributes:
+  value: fixedtpm|fixedparent|sensitivedataorigin|userwithauth|decrypt|sign
+  raw: 0x60072
+type:
+  value: rsa
+  raw: 0x1
+exponent: 65537
+bits: 2048
+scheme:
+  value: null
+  raw: 0x10
+scheme-halg:
+  value: (null)
+  raw: 0x0
+sym-alg:
+  value: null
+  raw: 0x10
+sym-mode:
+  value: (null)
+  raw: 0x0
+sym-keybits: 0
+rsa: c8cebe46344bbed17c39a497c3e5c53406be142ce741697641d940b77a835b3956c4ce0c5949688ff44a5d8ef847097e1870589ff4afcd401d2b7814b9a57ecc1f750b8a759b4e4f59915d8dda68c5463c8392870a59e21a02481e4d9b8d7ad27dd915850a587b6ff1a87fa98c578a0188e74c2731e39456c4e2e7f3158a878a294f82105a6ead9e397c15cd80c8b587c9a3f47513680cbe5f5fb5a0a41830566e5b70f312fa5e28fc780f45e72d4c8aa42fc2ea9d19e1068815493e2acda90cd6f7dabede223b494f916bd0c67682d4d5b4073b80954c0bab0ac612ae243f92c1d85ab3a7840d1d4aa7390f6155edb3341f229fbc015a8637d16230da03920f
+root@rpitest:~# INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:78 Running command: tpm2_load -C primary.ctx -u key.pub -r key.priv -c key.ctx
+tpm2_load -C primary.ctx -u key.pub -r key.priv -c key.ctx
+name: 000b0395380f392a3ef0773853ed245ed1a2ba94d26261d846268146f2f4de148cf0
+root@rpitest:~# INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:78 Running command: echo my message > message.dat
+echo my message > message.dat
+root@rpitest:~# INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:78 Running command: tpm2_sign -c key.ctx -g sha256 -o sig.rssa message.dat
+tpm2_sign -c key.ctx -g sha256 -o sig.rssa message.dat
+root@rpitest:~# INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:78 Running command: tpm2_verifysignature -c key.ctx -g sha256 -s sig.rssa -m message.dat
+.dat_verifysignature -c key.ctx -g sha256 -s sig.rssa -m message
+root@rpitest:~# echo result: $?
+result: 0
+PASSED
+------------------------------------------------------------------- live log teardown --------------------------------------------------------------------
+INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:112 Attempting a soft power off
+poweroff
+root@rpitest:~# poweroff
+...
+[   80.068761] reboot: Power down
+
+test_on_rpi4.py::TestResource::test_power_off_camera PASSED
+test_on_rpi4.py::TestResource::test_power_on_camera PASSED
+test_on_rpi4.py::TestResource::test_power_on_hdmi
+--------------------------------------------------------------------- live log call ----------------------------------------------------------------------
+INFO     imagehash:imagehash.py:79 video comparing snapshot test_booting_empty_ok.jpeg: snapshot f0f0f0f0f0f0f0f0, ref f0f0f0f0f0f0f0f0, diff: 0
+INFO     imagehash:imagehash.py:79 video comparing snapshot test_booting_rainbow_ok.jpeg: snapshot 3c3c3c1c1c1c1c1c, ref 3c3c3c1c1c1c1c1c, diff: 0
+INFO     imagehash:imagehash.py:79 video comparing snapshot test_booting_raspberries_ok.jpeg: snapshot c000000000000000, ref c000000000000000, diff: 0
+PASSED
+test_on_rpi4.py::TestResource::test_login_console_hdmi
+--------------------------------------------------------------------- live log setup ---------------------------------------------------------------------
+INFO     jumpstarter.testing.utils:utils.py:15 Waiting for login prompt
+INFO     jumpstarter.testing.utils:utils.py:21 Logged in
+--------------------------------------------------------------------- live log call ----------------------------------------------------------------------
+INFO     imagehash:imagehash.py:79 video comparing snapshot test_booted_ok.jpeg: snapshot c0c0000000000000, ref c0c0000000000000, diff: 0
+PASSED
+------------------------------------------------------------------- live log teardown --------------------------------------------------------------------
+INFO     /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/test_on_rpi4.py:test_on_rpi4.py:112 Attempting a soft power off
+INFO     jumpstarter.client.lease:lease.py:63 Releasing Lease c33b74ff-ad92-42a6-aa88-2c8a944a297c
+
+
+============================================================= 6 passed in 303.59s (0:05:03) ==============================================================
+```

examples/soc-pytest/jumpstarter_example_soc_pytest/exporter.yaml

@@ -0,0 +1,24 @@
+apiVersion: jumpstarter.dev/v1alpha1
+kind: ExporterConfig
+endpoint: grpc.jumpstarter.example.com:443
+token: xxxxx
+export:
+  # a DUTLink interface to the DUT
+  dutlink:
+    type: jumpstarter_driver_dutlink.driver.Dutlink
+    config:
+      storage_device: "/dev/disk/by-id/usb-SanDisk_3.2_Gen_1_5B4C0AB025C0-0:0"
+  # an HDMI to USB capture card
+  video:
+    type: jumpstarter_driver_ustreamer.driver.UStreamer
+    config:
+      args:
+        device: '/dev/v4l/by-path/pci-0000:00:14.0-usbv2-0:3:1.0-video-index0'
+        resolution: 1920x1080
+  # a USB camera pointing to the DUT
+  camera:
+    type: jumpstarter_driver_ustreamer.driver.UStreamer
+    config:
+      args:
+        device: '/dev/v4l/by-path/pci-0000:00:14.0-usbv2-0:4:1.0-video-index0'
+        resolution: 1280x720

examples/soc-pytest/jumpstarter_example_soc_pytest/image/Makefile

@@ -0,0 +1,56 @@
+all: prepare-image
+
+###############################################################################
+# Image preparation targets
+###############################################################################
+
+download-image:
+	scripts/download-latest-raspbian
+
+prepare-image: images/latest.raw mount
+	scripts/prepare-latest-raw
+	touch images/.prepared
+	umount mnt
+
+images/.prepared:
+	make prepare-image
+
+images/latest.raw.xz:
+	make download-image
+
+images/latest.raw: images/latest.raw.xz
+	xz -d -f -v -T0 -k $^
+	touch images/latest.raw
+	rm -f images/.prepared
+
+clean-image:
+	rm -f images/.prepared
+	rm -f images/latest.raw
+
+clean-images: clean-image
+	rm -rf images/download.raspberrypi.org
+	rm -rf images/latest.raw.xz
+
+clean: clean-image clean-images
+
+###############################################################################
+# Image manipulation targets
+###############################################################################
+
+mnt:
+	mkdir -p $@
+
+umount:
+	umount mnt || true
+
+mount: umount images/latest.raw mnt
+	guestmount -a images/latest.raw -m /dev/sda2 -m /dev/sda1:/boot/firmware -o allow_other --rw mnt
+
+
+###############################################################################
+# phony targets are targets which don't produce files, just for utility
+###############################################################################
+
+
+.PHONY: download-image prepare-image
+.PHONY: mount umount

examples/soc-pytest/jumpstarter_example_soc_pytest/image/README.md

@@ -0,0 +1,76 @@
+# Image preparation scripts
+
+This directory contains scripts to prepare the image for the example SOC test,
+running `make` should:
+
+* Download a minimal raspbian image
+* Inject the settings for the test to be performed (enable UART, setup basic password, tpm dtb)
+
+You will need guestmount installed, sudo permissions.
+
+fuse must be configured to enable `user_allow_other` in `/etc/fuse.conf`.
+
+
+```bash
+$ make
+make download-image
+make[1]: Entering directory '/home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/image'
+scripts/download-latest-raspbian
+https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2024-07-04/2024-07-04-raspios-bookworm-armhf-lite.img.xz
+--2024-10-17 12:12:43--  https://downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2024-07-04/2024-07-04-raspios-bookworm-armhf-lite.img.xz
+Resolving downloads.raspberrypi.org (downloads.raspberrypi.org)... 2a00:1098:80:56::2:1, 2a00:1098:80:56::1:1, 2a00:1098:82:47::1, ...
+Connecting to downloads.raspberrypi.org (downloads.raspberrypi.org)|2a00:1098:80:56::2:1|:443... connected.
+HTTP request sent, awaiting response... 200 OK
+Length: 523828628 (500M) [application/x-xz]
+Saving to: ‘./images/downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2024-07-04/2024-07-04-raspios-bookworm-armhf-lite.img.xz’
+
+downloads.raspberrypi.org/raspios_lite_armhf/ima 100%[=======================================================================================================>] 499.56M  77.9MB/s    in 6.6s
+
+2024-10-17 12:12:53 (75.3 MB/s) - ‘./images/downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2024-07-04/2024-07-04-raspios-bookworm-armhf-lite.img.xz’ saved [523828628/523828628]
+
+FINISHED --2024-10-17 12:12:53--
+Total wall clock time: 10s
+Downloaded: 1 files, 500M in 6.6s (75.3 MB/s)
+Latest image: ./images/downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2024-07-04/2024-07-04-raspios-bookworm-armhf-lite.img.xz
+Updating link from latest.raw.xz -> ./images/downloads.raspberrypi.org/raspios_lite_armhf/images/raspios_lite_armhf-2024-07-04/2024-07-04-raspios-bookworm-armhf-lite.img.xz
+make[1]: Leaving directory '/home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/image'
+xz -d -f -v -T0 -k images/latest.raw.xz
+images/latest.raw.xz (1/1)
+  100 %     499.6 MiB / 2,512.0 MiB = 0.199   102 MiB/s       0:24
+touch images/latest.raw
+rm -f images/.prepared
+umount mnt || true
+umount: /home/majopela/jumpstarter/examples/soc-pytest/jumpstarter_example_soc_pytest/image/mnt: not mounted.
+guestmount -a images/latest.raw -m /dev/sda2 -m /dev/sda1:/boot/firmware -o allow_other --rw mnt
+scripts/prepare-latest-raw
++ sudo sed -i 's/console=serial0,115200 console=tty1/console=serial0,115200/g' mnt/boot/firmware/cmdline.txt
++ cat mnt/boot/firmware/cmdline.txt
+console=serial0,115200 root=PARTUUID=d28ec40f-02 rootfstype=ext4 fsck.repair=yes rootwait quiet init=/usr/lib/raspberrypi-sys-mods/firstboot
++ cat
++ sudo tee mnt/boot/firmware/custom.toml
+# Raspberry Pi First Boot Setup
+[system]
+hostname = "rpitest"
+
+[user]
+name = "root"
+password = "changeme"
+password_encrypted = false
+
+[ssh]
+enabled = false
+
+[wlan]
+country = "es"
+
+[locale]
+keymap = "es"
+timezone = "Europe/Madrid"
++ cat
++ sudo tee -a mnt/boot/firmware/config.txt
+dtparam=spi=on
+dtoverlay=tpm-slb9670
+enable_uart=1
+touch images/.prepared
+umount mnt
+```

examples/soc-pytest/jumpstarter_example_soc_pytest/image/images/.gitignore

@@ -0,0 +1 @@
+*

examples/soc-pytest/jumpstarter_example_soc_pytest/image/images/.gitkeep

@@ -0,0 +1,22 @@
+#!/bin/bash
+set -e
+#LATEST_URL=$(wget -O /dev/null -o - --max-redirect=0 https://downloads.raspberrypi.org/raspios_lite_armhf_latest 2>/dev/null| sed -n "s/^Location: \(.*\) \[following\]$/\1/p")
+LATEST_URL=$(curl https://downloads.raspberrypi.org/raspios_lite_armhf_latest -v 2>&1 | sed -n "s/< location: \(.*\)\r$/\1/p")
+echo $LATEST_URL
+CACHE="./images"
+wget "${LATEST_URL}" -np -m -A '*img.xz' -c -P "${CACHE}"
+# use the latest compose image
+LATEST_IMG=$(ls -Art "${CACHE}/downloads.raspberrypi.org/raspios_lite_armhf/images"/*/*.img.xz | tail -n 1)
+
+echo "Latest image: ${LATEST_IMG}"
+
+# calculate full path to LATEST_IMG
+LATEST_IMG_FULLPATH=$(readlink -f ${LATEST_IMG})
+EXISTING_LINK=$(readlink "${CACHE}/latest.raw.xz" || true )
+# if the link has changed, update the link
+if [[ "${LATEST_IMG_FULLPATH}" != "${EXISTING_LINK}" ]]; then
+    echo "Updating link from latest.raw.xz -> ${LATEST_IMG}"
+    ln -fs "${LATEST_IMG_FULLPATH}" "${CACHE}/latest.raw.xz"
+else
+    echo "We are up-to-date."
+fi

examples/soc-pytest/jumpstarter_example_soc_pytest/image/scripts/download-latest-raspbian

@@ -0,0 +1,32 @@
+#!/bin/sh
+set -x
+# all output to serial port
+sudo sed -i 's/console=serial0,115200 console=tty1/console=serial0,115200/g' mnt/boot/firmware/cmdline.txt
+cat mnt/boot/firmware/cmdline.txt
+
+cat << EOF | sudo tee mnt/boot/firmware/custom.toml
+# Raspberry Pi First Boot Setup
+[system]
+hostname = "rpitest"
+
+[user]
+name = "root"
+password = "changeme"
+password_encrypted = false
+
+[ssh]
+enabled = false
+
+[wlan]
+country = "es"
+
+[locale]
+keymap = "es"
+timezone = "Europe/Madrid"
+EOF
+
+cat << EOF | sudo tee -a mnt/boot/firmware/config.txt
+dtparam=spi=on
+dtoverlay=tpm-slb9670
+enable_uart=1
+EOF