I have to NIC devices on the host:
# list Ethernet PCI devices to find out names
lspci -nn | grep Ethernet
# 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)
# 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
# to get <domain>.<bus>.<slot>.<function>
lspci -n -s 0000:04:00.0
# 04:00.0 0200: 10ec:8168 (rev 06)
And I want to pass through the device 0000:04:00.0 to the KVM ubuntu 20.04 virtual machine. The host may not see this device when VM is running. To bind PCI NIC to the guest I successfully followed instruction VFIO - "Virtual Function I/O". But unbinding from the guest to the host came more difficult.
I bind NIC device from the host to the guest manually in the following way:
# find Ethernet controller
ls -l /sys/class/net/ | grep pci
# enp3s0 -> ../../devices/pci0000:00/0000:00:1c.5/0000:03:00.0/net/enp3s0
# enp4s0 -> ../../devices/pci0000:00/0000:00:1c.7/0000:04:00.0/net/enp4s0
lspci -nn | grep Ethernet
# 04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)
# 05:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
# check the IOMMU grouping
for a in /sys/kernel/iommu_groups/*; do find $a -type l; done | sort --version-sort | grep '04:00\|05:00'
# /sys/kernel/iommu_groups/12/devices/0000:04:00.0
# /sys/kernel/iommu_groups/13/devices/0000:05:00.0
# find MAC addresses
for eth in enp4s0 enp5s0; do ifconfig $eth | grep ether | echo "MAC" "$eth:" $(awk '{print $2}'); done
# MAC enp4s0: 50:...
# MAC enp5s0: 18:...
# load the vfio-pci device driver
modprobe vfio_pci
# unbind from host
echo 0000:04:00.0 | sudo tee /sys/bus/pci/devices/0000:04:00.0/driver/unbind
# make available for guests
echo 10ec 8168 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
# IOMMU group for device 0000:04:00.0 is 12
# add libvirt-qemu to sudoers
sudo usermod -aG sudo libvirt-qemu
sudo chown libvirt-qemu:libvirt-qemu /dev/vfio/12
# look at what other devices are in the group to free it for use by VFIO
ls -l /sys/bus/pci/devices/0000:04:00.0/iommu_group/devices | awk '{print $9 $10 $11}'
# 0000:04:00.0->../../../../devices/pci0000:00/0000:00:1c.5/0000:04:00.0
# also add other devices if they belong to the same group (not needed in this case)
# make sure the device is not visible from host
ls -l /sys/class/net/ | grep enp4s0
Then I created new Ubuntu 20.04 VM using Virtual Machine Manager (virt-manager) to run on KVM.
I added new device to VM by editing its xml configuration in virt-manager during creation. In particular, <devices> section contains the following tag
<hostdev mode="subsystem" type="pci" managed="yes">
<driver name="vfio"/>
<source>
<address domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</source>
<mac address='50:...'/>
<address type="pci">
<zpci uid="0x0001" fid="0x00000000"/>
</address>
</hostdev>
Than I installed Ubuntu 20.04 in regular way. The system reboots properly, without deadlocks (black screen).
When I turn off the VM, I want to return the PCI NIC to the host. I did a research on forums but there is no clear instructions on how to do that.
If I reboot the host, all devices return to the host, so vfio binding is released. But how I can do that without host rebooting.
