Skip to content

Commit 8dedc74

Browse files
hyesoonhyesoon
authored
Merge pull request #248 from rahul7rajdn/apptainer
Apptainer for Vortex
2 parents a9050c4 + cc7e731 commit 8dedc74

8 files changed

Lines changed: 271 additions & 0 deletions

File tree

‎.gitignore‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,4 @@
22
/.vscode
33
*.cache
44
*.code-workspace
5+
miscs/apptainer/vortex.sif

‎docs/fpga_setup.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,12 @@ $ source /opt/xilinx/xrt/setup.sh
6363
$ source /tools/reconfig/xilinx/Vitis/2023.1/settings64.sh
6464
```
6565

66+
If you are using an apptainer,
67+
```
68+
$ export LD_LIBRARY_PATH=/opt/boost-1.66/lib/:/opt/openssl-1.1/lib/:$LD_LIBRARY_PATH
69+
```
70+
71+
6672
### Check Installed FPGA Platforms
6773
`platforminfo -l` which tells us the correct name of the platform installed on the current fpga node. It should be used for the `PLATFORM` variable below. Otherwise, if there is an error then there was an issue with the previous two commands.
6874

‎miscs/apptainer/README.md‎

Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
# Apptainer Build Process
2+
3+
Use the Slurm scheduler to request an interactive job on Flubber9
4+
```
5+
salloc -p rg-fpga --nodes=1 --ntasks-per-node=64 --mem=16G --nodelist flubber9 --time=01:00:00
6+
```
7+
8+
Go to `apptainer` directory
9+
10+
```
11+
$ pwd
12+
vortex/miscs/apptainer
13+
14+
$ apptainer build --no-https vortex.sif vortex.def
15+
16+
```
17+
18+
19+
To start the apptainer,
20+
```
21+
$ chmod +x run_apptainer.sh
22+
$ ./run_apptainer.sh
23+
```
24+
25+
26+
| Note: Set env variables in `run_apptainer.sh` accordingly.
27+
28+
29+
Inside the Apptainer,
30+
```
31+
# should show devices connected to machine on which you are running this command
32+
Apptainer> lsusb
33+
34+
```
35+
36+
37+
# Vortex Simulation inside Apptainer
38+
39+
Go to bind of vortex repo,
40+
```
41+
Apptainer> cd /home/vortex
42+
Apptainer> mkdir build
43+
Apptainer> cd build
44+
Apptainer> ../configure --xlen=32 --tooldir=$HOME/tools
45+
46+
47+
Skip the below 3 steps, If tools are already present in the $HOME/tools
48+
Apptainer> sed -i 's/\btar /tar --no-same-owner /g' ci/toolchain_install.sh
49+
Apptainer> ./ci/toolchain_install.sh --all
50+
Apptainer> sed -i 's/\btar --no-same-owner /tar /g' ci/toolchain_install.sh
51+
52+
53+
Apptainer> source ./ci/toolchain_env.sh
54+
Apptainer> verilator --version
55+
```
56+
57+
58+
### Running SIMX, RTLSIM and XRTSIM
59+
```
60+
Compile the Vortex codebase
61+
Apptainer> make -s
62+
63+
Run the programs by specifying the appropriate driver as shown below:
64+
65+
SIMX
66+
Apptainer> ./ci/blackbox.sh --cores=2 --app=demo --driver=simx
67+
68+
RTLSIM
69+
Apptainer> ./ci/blackbox.sh --cores=2 --app=demo --driver=rtlsim
70+
71+
XRTSIM
72+
Apptainer> ./ci/blackbox.sh --cores=2 --app=demo --driver=xrt
73+
```
74+
75+
76+
### Vortex Bitstream on FPGA
77+
78+
Common Commands
79+
```
80+
Apptainer> source /opt/xilinx/xrt/setup.sh
81+
Apptainer> source /tools/reconfig/xilinx/Vitis/2023.1/settings64.sh
82+
83+
Apptainer> platforminfo -l
84+
85+
```
86+
87+
##### Building Vortex Bitstream
88+
89+
```
90+
Apptainer> pwd
91+
/home/vortex/build
92+
93+
Apptainer> source ci/toolchain_env.sh
94+
95+
Apptainer> verilator --version
96+
Verilator 5.026 2024-06-15 rev v5.026-43-g065f36ab5
97+
98+
Apptainer> cd hw/syn/xilinx/xrt
99+
Apptainer> PREFIX=test1 PLATFORM=xilinx_u50_gen3x16_xdma_5_202210_1 TARGET=hw NUM_CORES=1 make > build_u50_hw_1c.log 2>&1 &
100+
Creates ../test1_xilinx_u50_gen3x16_xdma_5_202210_1_hw/bin/vortex_afu.xclbin
101+
```
102+
103+
##### Running Vortex Bitstream on FPGA
104+
```
105+
Apptainer> cd ../../../../
106+
Apptainer> pwd
107+
/home/vortex/build
108+
109+
Apptainer> make -C runtime/ clean
110+
111+
Apptainer> FPGA_BIN_DIR=hw/syn/xilinx/xrt/test1_xilinx_u50_gen3x16_xdma_5_202210_1_hw/bin TARGET=hw PLATFORM=xilinx_u50_gen3x16_xdma_5_202210_1 ./ci/blackbox.sh --driver=xrt --app=demo
112+
113+
Verify following line being printed:
114+
info: device name=xilinx_u50_gen3x16_xdma_base_5, memory_capacity=0x200000000 bytes, memory_banks=32.
115+
```

‎miscs/apptainer/boost-package.deb‎

11.7 MB
Binary file not shown.

‎miscs/apptainer/install_boost_openssl.sh‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
#!/bin/bash
2+
3+
set -e # Exit on error
4+
echo "Installing Boost and OpenSSL from .deb packages"
5+
6+
# Extract and install Boost
7+
echo "Extracting Boost..."
8+
ar -x boost-package.deb
9+
ls
10+
mkdir -p /tmp/boost-extract
11+
tar --zstd --no-same-owner -xvf data.tar.zst -C /tmp/boost-extract
12+
ls -l /tmp/boost-extract/
13+
cp -r /tmp/boost-extract/opt/boost-1.66/ /opt/
14+
ls -l /opt/boost-1.66/
15+
rm -rf data.tar
16+
17+
# Extract and install OpenSSL
18+
echo "Extracting OpenSSL..."
19+
ar -x openssl-package.deb
20+
mkdir -p /tmp/openssl-extract
21+
tar --zstd --no-same-owner -xvf data.tar.zst -C /tmp/openssl-extract
22+
ls -l /tmp/openssl-extract/
23+
cp -r /tmp/openssl-extract/opt/openssl-1.1/ /opt/
24+
ls -l /opt/openssl-1.1/
25+
rm -rf data.tar
26+
27+
28+
29+
echo "Boost and OpenSSL installation complete!"

‎miscs/apptainer/openssl-package.deb‎

3.24 MB
Binary file not shown.

‎miscs/apptainer/run_apptainer.sh‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
#!/bin/bash
2+
3+
# Set environment variables with fallback default values.
4+
# The :- means if the variable CONTAINER_IMAGE is not set or is empty, then use the default value vortex.sif
5+
export GIT_REPO_PATH=${GIT_REPO_PATH:-~/USERSCRATCH/vortex}
6+
export CONTAINER_IMAGE=${CONTAINER_IMAGE:-vortex.sif}
7+
export DEVNULL_BIND=${DEVNULL_BIND:-~/USERSCRATCH/devnull}
8+
export VORTEX_TOOLCHAIN_PATH=${VORTEX_TOOLCHAIN_PATH:-~/USERSCRATCH/tools}
9+
10+
# Launch the Apptainer container with the bind mount
11+
apptainer shell --fakeroot --cleanenv --writable-tmpfs \
12+
--bind /dev/bus/usb,/sys/bus/pci \
13+
--bind /projects:/projects \
14+
--bind /lib/firmware:/lib/firmware \
15+
--bind /opt/xilinx/:/opt/xilinx/ \
16+
--bind /tools:/tools \
17+
--bind /netscratch:/netscratch \
18+
--bind "$VORTEX_TOOLCHAIN_PATH":/home/tools \
19+
--bind "$GIT_REPO_PATH":/home/vortex \
20+
"$CONTAINER_IMAGE"

‎miscs/apptainer/vortex.def‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
Bootstrap: docker
2+
From: ubuntu:22.04
3+
4+
%files
5+
install_boost_openssl.sh .
6+
openssl-package.deb .
7+
boost-package.deb .
8+
9+
%post
10+
echo "Setting up the environment..."
11+
12+
touch /var/log/apt/term.log
13+
if [ -f /var/log/apt/term.log ]; then
14+
chown root:adm /var/log/apt/term.log || true
15+
fi
16+
17+
# Set environment variable to avoid interactive prompts during installation
18+
export DEBIAN_FRONTEND=noninteractive
19+
echo 'APT::Sandbox::User "root";' > /etc/apt/apt.conf.d/no-sandbox
20+
mkdir -p /netscratch
21+
22+
# Update and install base dependencies (including wget here)
23+
apt-get update
24+
apt-get install -y libc6 fakeroot wget git vim
25+
26+
# Fix dependency issues
27+
apt-get install -f -y || true
28+
dpkg --configure -a || true
29+
30+
# Reinstall problematic packages
31+
apt-get remove --purge -y dbus fontconfig-config || true
32+
apt-get install -y dbus fontconfig-config || true
33+
34+
rm -f /var/lib/dpkg/lock-frontend
35+
rm -f /var/lib/dpkg/lock
36+
rm -f /var/cache/apt/archives/lock
37+
38+
# Adjust permissions for dbus helper
39+
chown -R root:root /usr/lib/dbus-1.0/dbus-daemon-launch-helper || true
40+
41+
# Install necessary packages
42+
apt-get install -y fontconfig libfontconfig1 libpangoft2-1.0-0 libbluray2 \
43+
libpangocairo-1.0-0 libavformat58 libpango-1.0-0 \
44+
openjdk-11-jre-headless libcairo2 librsvg2-2 librsvg2-common \
45+
openjdk-11-jre-zero libtheora0 libavcodec58 libcairo-gobject2 \
46+
ca-certificates-java libchromaprint1 software-properties-common perl-modules bzip2 \
47+
unzip zlib1g-dev libtinfo5 g++ usbutils gawk bison gcc make tar python3.9 locales zstd uuid-dev ccache || true
48+
49+
ln -s /usr/bin/python3 /usr/bin/python
50+
51+
locale-gen en_US.UTF-8
52+
update-locale LANG=en_US.UTF-8 LANGUAGE=en_US.UTF-8 LC_ALL=en_US.UTF-8
53+
54+
# Run dpkg to ensure configuration is complete
55+
dpkg --configure -a || true
56+
57+
# Clean up apt cache
58+
apt-get clean
59+
rm -rf /var/lib/apt/lists/*
60+
61+
echo "Setting up the environment..."
62+
63+
# Install CMake 3.26
64+
echo "Installing CMake 3.26..."
65+
wget https://github.com/Kitware/CMake/releases/download/v3.26.0/cmake-3.26.0-linux-x86_64.sh -O /tmp/cmake.sh
66+
chmod +x /tmp/cmake.sh
67+
mkdir -p /opt/cmake
68+
/tmp/cmake.sh --skip-license --prefix=/opt/cmake
69+
ln -sf /opt/cmake/bin/cmake /usr/local/bin/cmake
70+
rm -f /tmp/cmake.sh
71+
72+
# Copy the boost and openssl installation script into the container
73+
cp install_boost_openssl.sh /tmp/
74+
75+
# Execute the installation script
76+
chmod +x /tmp/install_boost_openssl.sh
77+
/tmp/install_boost_openssl.sh
78+
79+
# Clean up
80+
rm -f /tmp/install_boost_openssl.sh
81+
82+
%environment
83+
export LANG=en_US.UTF-8
84+
export LANGUAGE=en_US.UTF-8
85+
export LC_ALL=en_US.UTF-8
86+
export HOME=/home
87+
export LD_LIBRARY_PATH=/opt/boost-1.66/lib:/opt/openssl-1.1/lib:$LD_LIBRARY_PATH
88+
89+
%runscript
90+
exec /bin/bash
91+
92+
%labels
93+
Author Rahul, Udit
94+
Version 2.0
95+
96+
%test
97+
locale
98+
lsusb || echo "lsusb not available"
99+
ls -l /dev/null
100+
echo "Container test successful!"

0 commit comments

Comments
 (0)