Virtualized hardware acceleration on QubesOS using Intel Arc B50

Setting up the Arc Pro B50

Update the Arc Pro firmware

The Arc Pro B50, out of factory, does not include a firmware that supports SR-IOV. We thus have to update it. Unfortunately, it looks like newer version of the firmware artifically limits to VF count to 2 rather than 12, when the feature was first supported. You might want to flash to older version, in this case.

Driver overview for the B50:

The last driver in the list is still the one to get to have access to all Virtual Functions. The easy way to flash the new drivers is by installing, in a windows environment, the Intel Arc Pro Graphic driver of the version you desire. Unfortunately, this requires running windows on bare-metal to flash safely.

There is a way to update the firmware using fwupd, but this guide has not documented this way yet on QubesOS.

Determine the device address

$ lspci -vv | grep B50
06:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Arc Pro B50] (prog-if 00 [VGA controller])

For the rest of the guide, I will reference the B50 with 0000:06:00.0. Update the snippets accordingly.

$ lspci -vv -s 06:00.0
06:00.0 VGA compatible controller: Intel Corporation Battlemage G21 [Arc Pro B50] (prog-if 00 [VGA controller])
    Subsystem: Intel Corporation Device 1114
    Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Latency: 0, Cache Line Size: 64 bytes
    Interrupt: pin ? routed to IRQ 228
    IOMMU group: 13
    Region 0: Memory at 13e0c000000 (64-bit, prefetchable) [size=16M]
    Region 2: Memory at 13400000000 (64-bit, prefetchable) [size=16G]
    Expansion ROM at cdc00000 [disabled] [size=2M]
    Capabilities: <access denied>
    Kernel driver in use: xe
    Kernel modules: xe

If everything went well the last two lines should point to the xe driver.

Query max Virtual Functions

$ cat /sys/bus/pci/devices/0000:06:00.0/sriov_totalvfs
2

Boot process

Hardware

Not all hardware will support the SR-IOV use case. The following needs to be enabled in BIOS:

In my case, I have a Supermicro X10-based system, which means ResizeBar is not supported. To unlock, I thus leveraged xCurio0’s ReBarUEFI to patch in ResizeBar support. How to do this is out-of-scope of this guide.

Kernel

This also requires a newer kernel. At the time of writing this guide, kernel 6.19 works, but kernel 6.17 theoretically includes SR-IOV support in xe drivers

Disabling auto probing

There is a currently a bug with Xen, where unbinding xe from virtual functions after creation leads to a kernel crash. To mitigate this issue, we can disable automatic probing by leveraging the sriov_drivers_autoprobe sysfs parameter:

echo 0 | sudo tee /sys/bus/pci/devices/"$DEVICE"/sriov_drivers_autoprobe

This will ensure that the virtual function will be able to attach the device when starting the VM.

That said, the following udev rule can be created to do this automatically:

ACTION="add", SUBSYSTEM="pci", ATTR{sriov_totalvfs}=="*", ATTR{sriov_drivers_autoprobe}="0"

Enable virtual function creation

Due to how the driver works, when the virtual functions are created, whatever VRAM is not currently in use will be split up between the functions. Thus, to keep some VRAM for the host, we have to allocate some memory, before creating the VFs. There isn’tan elegant way to do so, so this guide proposes the use of vramfs to pre-allocate VRAM for the host, then drop the vram filesystem after creation of the virtual functions.

Unfortunately, vramfs is not packaged in Fedora, so it is provided by myself via my qubes-pkgs repo, and installable through dnf using this package.

To facilitate setting up the virtual functions, I took vitabis’ script and wrapped it in a udev rule. So, on dom0, you can create the following file under /usr/local/bin/spawn_xe_vfs.sh:

#!/bin/bash

###################################################################
# Helper script to automate the initialization of Virtual Functions
# The idea is to quickly allocate memory space on the Arc Pro GPU,
# before the Virtual Functions are created. After VF creation,
# this memory is released and will remain available to the PF.
###################################################################

### VARIABLES FOR USER TO SET (REQUIRED)
# The Physical Function's address (eg from lspci, domain:bus:device.function)
DEVICE="$1"
# Number of Virtual Functions to create
NUM_VFS="$2"

### VARIABLES FOR USER TO OVERRIDE (OPTIONAL)
# Amount of memory that will remain available to the Physical Function after 
# Virtual Functions have been created (in KB/MB/GB).
VRAMFS_SIZE="$3"
# Entry path into the vramfs filesystem
VRAMFS_PATH="/tmp/vram"
# Delay until the vramfs filesystem is mounted (in seconds)
DELAY_VFS=3

# Create vramfs storage of requested size.
if [ ! -e "$VRAMFS_PATH" ]; then
    /usr/bin/mkdir $VRAMFS_PATH
fi
/usr/bin/vramfs $VRAMFS_PATH $VRAMFS_SIZE &

# Wait for the filesystem to mount (in the background) before spawning VFs
sleep $DELAY_VFS

# Create Virtual Functions
echo "creating vfs."
/usr/bin/echo $NUM_VFS > "/sys/bus/pci/devices/"$DEVICE"/sriov_numvfs"

# Tear down the vramfs filesystem
echo "terminating vramfs."

Make it executable:

chmod +x /usr/local/bin/spawn_xe_vfs.sh

Finally, create the udev rule under /etc/udev/rules.d/81-enable-sriov-numvfs.rules:

ACTION="add", SUBSYSTEM=="pci", ATTR{subsystem_device}=="0x1114", ATTR{subsystem_vendor}=="0x8086", RUN+="/usr/local/bin/spawn_xe_vfs.sh %k 2 1GB"

This rule, when adding the Intel Arc B50, executes spawn_xe_vfs.sh and feeds the device address, tells the script to create 2 virtual functions, while pre-allocating 1GB of VRAM to the host. You can adjust these arguments depending on your needs, and which firmware you used.

Assign VF to VM

After this, you should be able to assign the virtual functions to the VM like any other PCI device. You should use, at least, the kernel version used in dom0. The VM should also be an HVM, like any other VM you attach devices to.

References

Known issues

Created . Edited .