1 Problem
My goal is that all LAN traffic is routed through a virtual machine (VM) acting as a router and firewall. Other VMs on the same hypervisor should also be routed through the firewall VM. See the following network sketch. Physical hosts connected to the trunk bridge can ping each other successfully, but VMs could not be reached yet.
2 Details
Network configuration on the hypervisor:
# =======
# Bridges
# =======
# ---------------------
# VLAN bridge br-vlan-3
# ---------------------
ip link add name br-vlan-3 type bridge
ip addr add 10.66.3.11/24 dev br-vlan-3
ip link set dev br-vlan-3 up
# ---------------------
# Trunk bridge Upstream
# ---------------------
ip link add name br-trunk-up type bridge vlan_filtering 1 vlan_default_pvid 0
ip link set br-trunk-up up
# ---------------
# Trunk bridge VM
# ---------------
ip link add name br-trunk-vm type bridge vlan_filtering 1 vlan_default_pvid 0
ip link set br-trunk-vm up
# =====
# Hosts
# =====
# Set physical devices up
ip link set enp5s0 up
ip link set enp9s0f0 up
ip link set enp9s0f1 up
ip link set enp9s0f2 up
ip link set enp9s0f3 up
# ----------------------
# desktop.3 br-vlan-3
# ----------------------
ip link add link enp9s0f1 name enp9s0f1.3 type vlan id 3
ip link set dev enp9s0f1.3 master br-vlan-3
ip link set enp9s0f1.3 up
# --------------------------
# desktop.100 br-trunk-vm
# --------------------------
ip link add link enp9s0f1 name enp9s0f1.100 type vlan id 100
ip link set enp9s0f1.100 master br-trunk-vm
ip link set enp9s0f1.100 up
bridge vlan add vid 100 dev enp9s0f1.100 master
bridge vlan add vid 100 dev br-trunk-vm self
bridge vlan add vid 100 dev fw.100 pvid 100
Network map
=======
DESKTOP
=======
localhost:user ~ $ ip a
2: enp34s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 00:d8:61:c4:b1:af brd ff:ff:ff:ff:ff:ff
4: br-vlan-100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 1a:b1:36:43:f0:e4 brd ff:ff:ff:ff:ff:ff
inet 10.66.100.44/24 scope global br-vlan-100
valid_lft forever preferred_lft forever
7: enp.100@enp34s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-vlan-100 state UP group default qlen 1000
link/ether 00:d8:61:c4:b1:af brd ff:ff:ff:ff:ff:ff
localhost:~ # cat /proc/net/vlan/enp.100
enp.100 VID: 100 REORDER_HDR: 1 dev->priv_flags: 1221
total frames received 0
total bytes received 0
Broadcast/Multicast Rcvd 0
total frames transmitted 287
total bytes transmitted 53964
Device: enp34s0
INGRESS priority mappings: 0:0 1:0 2:0 3:0 4:0 5:0 6:0 7:0
EGRESS priority mappings:
==========
HYPERVISOR
==========
bridge name bridge id STP enabled interfaces
br-vlan-100 8000.1ab13643f0e4 no enp.100
[root@fedora system]# ip a
4: enp9s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether a0:36:9f:5d:09:19 brd ff:ff:ff:ff:ff:ff
13: br-trunk-vm: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 56:7e:22:6a:23:2d brd ff:ff:ff:ff:ff:ff
16: enp9s0f1.100@enp9s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-trunk-vm state UP group default qlen 1000
link/ether a0:36:9f:5d:09:19 brd ff:ff:ff:ff:ff:ff
21: fw.100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-trunk-vm state UNKNOWN group default qlen 1000
link/ether fe:54:00:3e:97:72 brd ff:ff:ff:ff:ff:ff
[root@fedora system]# brctl show dev br-trunk-vm
bridge name bridge id STP enabled interfaces
br-trunk-vm 8000.567e226a232d no enp9s0f1.100
fw.100
fw.103
fw.3
===============
VIRTUAL MACHINE
===============
[root@fedora system]# bridge vlan show
port vlan-id
br-trunk-vm 100
enp9s0f1.100 100
virbr0 1 PVID Egress Untagged
fw.100 100 PVID Egress Untagged
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 52:54:00:3e:97:72 brd ff:ff:ff:ff:ff:ff
inet 10.66.100.10/24 scope global enp1s0
valid_lft forever preferred_lft forever
Libvirt XML network configuration of the firewall VM:
<interface type='bridge'>
<mac address='52:54:00:3e:97:72'/>
<source bridge='br-trunk-vm'/>
<target dev='fw.100'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
3 Attempts
I tried configuring both a tagged and untagged NIC on the firewall VM. For the tagged NIC I created a VLAN bridge on the firewall VM and created a virtual tap device from the virtual tap fw.100 with the VLAN 100: [email protected]. Then I set the trunk port of [email protected] to master: bridge vlan add vid 100 dev [email protected] master. For the untagged NIC approach I configured no VLAN on the firewall VM and set its trunk port to PVID 100 untagged bridge vlan add vid 100 dev fw.100 pvid 100 untagged. In both cases I could not ping between the desktop host and the firewall VM.
This issue happens only with the libvirt KVM/QEMU virtual machine. Two physical VLAN tagged hosts connected to the hypervisor's trunk bridge as master can ping each other.
4 Update
It looks like that libvirt does not support Linux bridges with VLANs. According to following article the VMs VLAN stops working if there is a second virtual device on the hypervisor with the same VLAN: https://alesnosek.com/blog/2015/09/07/bridging-vlan-trunk-to-the-guest/
Alternatively openvswitch can be used, yet using a bridge for each VM and physical host is a better approach. The hypervisor then allows forwarding the VMs' and hosts' bridges to the firewall VM's bridge and the remaining routing is then done inside the firewall's VM.
