Gaming VM on Qubes Using Bazzite

My main workstation now uses QubesOS rather than Proxmox. This workstation has two GPUs and two USB controllers. I wanted to be able to assign the more powerful GPU to a Gaming Qubes, but Qubes doesn’t support that well. On top of that, the Sapphire Pulse 6700XT GPU that I have has the AMD reset bug. To seamlessly be able to shutdown and assign the GPU to another VM (say my video editing environment), I applied a workaround that involved flashing a BIOS from another (working) 6700XT model (the Sapphire Nitro+ 6700XT). This solved the issue, at the cost of having to apply the original romfile on VM start. While this is easy on Proxmox (via romfile options), it isn’t so easy on Qubes.

I have ressources down at the bottom one can refer to for more information. In any case, lets start the guide.

A note: QubesOS explicitely does not support GPU Passthrough as these devices are notoriously bad at security. I’ve done my threat modelling, you should do yours to make sure GPU passthrough doesn’t expose you. The team is working on safe GPU acceleration, but it looks like it stalled.

Setup Bazzite

qvm-run -p $vm 'cat /path/to/iso' > bazzite.iso
qvm-start --cdrom=dom0:/path/to/iso bazzite

Flash good bios on Sapphire Pulse 6700XT

(this is copy-pasted almost verbatim from a level1techs.com forum post by Mechanical)

I did this for the Sapphire Pulse 6700XT, I make no guarantees this will work on other cards, not even on this card if you have one

First off you need to check your card’s memory vendor and you need amdvbflash. I used version 4.71. You should be able to find your memory vendor by checking GPU-Z in Windows using the stock VBIOS on your current card. Pass it through to your WIndows guest as usual and check GPU-Z for further information.

The memory vendor and chip needs to be one of the following:

If it isn’t you’re taking a gigantic risk flashing anything.

Next up is dumping the VBIOS. You can also do that through GPU-Z, but I’d do it through Linux as well to ensure the VBIOS is matching. You need to turn off VFIO anyway to flash a VBIOS. Comment out vfio-pci binding the GPU at boot like in /etc/modprobe.d using # and rebuild initramfs using something like mkinitcpio -P and rebooting. The GPU needs to be taken by amdgpu module.

The command for dumping the BIOS using amdvbflash is:

./amdvbflash -s 0 MyOriginalVBIOS.rom

You can also just dump it through sysbus:

cat /sys/bus/pci/devices/0000:0X:00.0/rom

Where the X is the hexadecimal digit for where your GPU is located. You can find it using lspci -vvv . You probably know yours already. Dump multiple version of your VBIOS and ensure their checksum are the same so that you don’t sit there with a corrupted VBIOS backup and put it on another machine/cloud to ensure you have it around.

As everyone points in every guide about these kind of things, but it needs to be said anyway:

Flashing any VBIOS, especially cross-vendor like this, will 100% destroy your warranty and you might sit there with a brick.

Use caution and remember it is your own fault if it never boots up ever again. You should have dual-BIOS possibilities on your card to be safer. Single-BIOS is very dangerous. You have been warned. So to flash the thing, get the working VBIOS from here: VGA Bios Collection: Sapphire RX 6700 XT 12 GB | TechPowerUp 48 The MD5 for this should be bbcf8fd1e226609094cd2283b3ea2259

Next up you need to flash the thing.

The -p 0 here means card 0, make sure you’re using the correct card and not your host’s card!!!

./amdvbflash -i # Check that 0 is the card you want to flash first!
./amdvbflash -fs -fp -fv -p 0 ./249630.rom # This is the actual flashing

That will essentially flash and change the IDs of the card to match the ROM.

You can verify that the ROM was correctly flashed by using:

./amdvbflash -v 0 ./249630.rom

If everything looks ok, shut down the system completely, cold boot and pray things work out. If it boots up in Linux now without any issues you should re-enable the VFIO-PCI module so it binds the card on the next boot. Don’t forget to rebuild initramfs.

Lastly, use your original VBIOS and pass that to the guest. That way the card will use the correct clocks for your card and make it run like it used to.

I would NOT recommend using the Sapphire RX 6700 XT on the host, use your original VBIOS so it runs correctly.

Setup PCI Passthrough

In my configuration, I pass USB device + GPU + NVMe (containing game files), while also loading the original GPU ROM on the guest. Unlike Proxmox, Qubes does not support this, even though Xen supports setting a custom ROM file. A creative (read: hacky) solution by chriswoope exists to introduce the romfile, but it involves adding the romfile to the linux stubdomain and making changes to vchan-socket-proxy to add the necessary arguments when adding the GPU as a device. * Add GPU bios to stubdomain using following script that can be created under /usr/local/bin/add-gpubios-stubdomain.sh:

#!/usr/bin/bash

gpubio=$1

for stubdom in /usr/libexec/xen/boot/qemu-stubdom-linux-rootfs /usr/libexec/xen/boot/qemu-stubdom-linux-full-rootfs; do
    dir="$(mktemp -d)"
    cd "$dir"
    if ! test -e "$stubdom.orig"; then
        mv "$stubdom" "$stubdom.orig"
    fi
    zcat "$stubdom.orig"|cpio -idm
    cp -a "${gpubios:=/usr/libexec/xen/boot/gpubios}" ./share/gpubios
    find .|cpio -o -c|gzip -9 > "$stubdom"
done
mv /usr/bin/vchan-socket-proxy /usr/bin/vchan-socket-proxy.orig
qubes-dom0-update nmap-ncat
#!/usr/bin/bash

vchan-socket-proxy.orig "$1" "$2" "$3" "$4" "$5.internal" &
ncat -k --listen --unixsock "$5" --sh-exec "sed -u -e 's|\"hostaddr\":\"0000:07:00.0\"|\"hostaddr\":\"0000:07:00.0\",\"romfile\":\"/share/gpubios\"|'|ncat --unixsock $5.internal"
chmod +x /usr/bin/vchan-socket-proxy /usr/local/bin/add-gpubios-stubdomain.sh
add-gpubios-stubdomain.sh /path/to/rom
qvm-pci list
qvm-pci attach bazzite (address of GPU) --persistent -o no-strict-reset=true -o permissive=true
qvm-pci attach bazzite (address of GPU audio) --persistent -o no-strict-reset=true -o permissive=true
qvm-pci attach bazzite (address of USB controller) --persistent -o no-strict-reset=true -o permissive=true
qvm-features bazzite video-model none

Configuring Steam storage and ModOrganizer2.

Since I am using an nvme drive for the game files, I setup this up as a btrfs drive using the included disk manager. In terminal, on Bazzite’s desktop, I setup two subvolumes as steam and mods. The steam submodule is mounted using /etc/fstab under /home/bazzite/.local/share/steam while mods is mounted under /home/bazzite/Mods. The latter contains mod files used by ModOrganizer2.

ModOrganizer2 can be setup using Furglitch’s modorganiser2-linux-installer. The following should be noted:

sudo ln -s /usr/bin/qtpaths6 /usr/local/bin/qtpaths

Other notes

Other ressources

* Edit /etc/default/grub to include rd.qubes.hide_pci, iommu=pt, video=vesafb:off video=efifb:off video=simplefb:off nofb initcall_blacklist=sysfb_init (all the frame buffer stuff is a precaution since I noticed without it I would get text on the monitor connected to the gpu indicating memory being initialized.) See Explaining CSM, efifb=off, and Setting the * Boot GPU Manually - The Passthrough POST for some background on csm vs uefi and how it affects gpu passthrough.
* Edit /usr/share/qubes/templates/libvirt/devices/pci.xml to pass a techpower rom file (didn’t seem to help). For those who care, I filtered on vm.name and device.libvirt_name=‘pci_0000_05_00_0’.
* Edit /usr/share/qubes/templates/libvirt/devices/pci.xml to set rom bar=off (didn’t see to help)
* Edit /usr/share/qubes/templates/libvirt/xen.xml to disable the cpu hypervisor feature (no noticeable affect)
* Create an sh file to change bar0 to 32GB and bar2 to 8MB. This is apparently necessary for Windows to avoid error 43 for whatever reason. Enabling or disabling rebar in bios does not seem to change behavior (may change performance) [SOLVED] 7900 XTX Code 43 or How to get 7900 XTX to work on VFIO - Virtualization - Level1Techs Forums
* Create an sh file to remount the gpu (remove and rescan).
* Attach the GPU with permissive and no-strict-reset. Didn’t seem to change behavior.
* Enable everything related to virtualization in my motherboard bios (proart x670e-creator)
* Disable motherboard and cpu ASPM in my motherboard bios. Didn’t seem to change behavior.
Created . Edited .