Skip to content

Latest commit

 

History

History
466 lines (351 loc) · 12.7 KB

Networking.asciidoc

File metadata and controls

466 lines (351 loc) · 12.7 KB

Networking in OpenQA

Table of Contents
Important
 This overview is valid only when using the QEMU backend!

The networking type used is controlled by the NICTYPE variable. If unset or empty NICTYPE defaults to user, i.e. QEMU user networking which requires no further configuration.

For more advanced setups or tests that require multiple jobs to be in the same networking the TAP or VDE based modes can be used.

QEMU User Networking

With QEMU user networking each jobs gets its own isolated network with TCP and UDP routed to the outside. DHCP is provided by QEMU. The MAC address of the machine can be controlled with the NICMAC variable. If not set, it is 52:54:00:12:34:56.

TAP Based Network

os-autoinst can connect QEMU to TAP devices of the host system to leverage advanced network setups provided by the host by setting NICTYPE=tap.

The TAP device to use can be configured with the TAPDEV variable. If not defined, it is automatically set to "tap" + ($worker_instance - 1), i.e. worker1 uses tap0, worker 2 uses tap1 and so on.

For multiple networks per job (see NETWORKS variable), the following numbering scheme is used:

worker1: tap0 tap64 tap128 ...
worker2: tap1 tap65 tap129 ...
worker3: tap2 tap66 tap130 ...
...

The MAC address of each virtual NIC is controlled by the NICMAC variable or automatically computed from $worker_id if not set.

In TAP mode the system administrator is expected to configure the network, required internet access, etc. on the host manually.

VDE Based Network

Virtual Distributed Ethernet provides a software switch that runs in user space. It allows to connect several QEMU instances without affecting the system’s network configuration.

The openQA workers need a vde_switch instance running. The workers reconfigure the switch as needed by the job.

Basic, Single Machine Tests

To start with a basic configuration like QEMU user mode networking, create a machine with the following settings:

  • VDE_SOCKETDIR=/run/openqa

  • NICTYPE=vde

  • NICVLAN=0

Start the switch and user mode networking:

systemctl enable --now openqa-vde_switch
systemctl enable --now openqa-slirpvde

With this setting all jobs on the same host would be in the same network and share the same SLIRP instance.

Multi Machine Tests Setup

The section provides one of the ways for setting up the openQA environment to run tests that require network connection between several machines (e.g. client — server tests).

The example of the configuration is applicable for openSUSE and will use Open vSwitch for virtual switch, firewalld (or SuSEfirewall2 for older versions) for NAT and wicked as network manager. Keep in mind that a firewall is not strictly necessary for operation. The operation without firewall is not covered in all necessary details in this documentation.

Note
 Another way to setup the environment with iptables and firewalld is described on the Fedora wiki.

Set Up Open vSwitch

Compared to VDE setup, Open vSwitch is slightly more complicated to configure, but provides a more robust and scalable network.

  • Install and Run Open vSwitch:

zypper in openvswitch
systemctl enable --now openvswitch

  • Install and configure os-autoinst-openvswitch.service:

Note
 os-autoinst-openvswitch.service is a support service that sets the vlan number of Open vSwitch ports based on NICVLAN variable - this separates the groups of tests from each other. The NICVLAN variable is dynamically assigned by the OpenQA scheduler. Install, start and enable the service: 
zypper in os-autoinst-openvswitch
systemctl enable --now os-autoinst-openvswitch

The service os-autoinst-openvswitch.service uses br0 bridge by default. As it might be used by KVM already it is suggested to configure br1 instead:

# /etc/sysconfig/os-autoinst-openvswitch
OS_AUTOINST_USE_BRIDGE=br1

  • Create the virtual bridge br1:

ovs-vsctl add-br br1

Configure Virtual Interfaces

  • Add a tap interface for every multi-machine worker instance:

Note
 Create as many interfaces as needed for a test. The instructions are provided for three interfaces tap0, tap1, tap2 to be used by worker@1, worker@2, worker@3 worker instances. The TAP interfaces have to be owned by the _openqa-worker user for the openQA worker instances to be able to access them.

To create tap interfaces automatically on startup, add appropriate configuration files to the /etc/sysconfig/network/ directory. Files have to be named as ifcfg-tap<N>, replacing <N> with the number for the interface, such as 0, 1, 2 (e.g. ifcfg-tap0, ifcfg-tap1):

# /etc/sysconfig/network/ifcfg-tap0
BOOTPROTO='none'
IPADDR=''
NETMASK=''
PREFIXLEN=''
STARTMODE='auto'
TUNNEL='tap'
TUNNEL_SET_GROUP='nogroup'
TUNNEL_SET_OWNER='_openqa-worker'

Symlinks can be used to reference the same configuration file for each tap interface.

  • Add the bridge config with all tap devices that should be connected to it. The file has to be located in the /etc/sysconfig/network/ directory. File name is ifcfg-br<N>, where <N> is the id of the bridge (e.g. 1):

# /etc/sysconfig/network/ifcfg-br1
BOOTPROTO='static'
IPADDR='10.0.2.2/15'
STARTMODE='auto'
OVS_BRIDGE='yes'
OVS_BRIDGE_PORT_DEVICE_1='tap0'
OVS_BRIDGE_PORT_DEVICE_2='tap1'
OVS_BRIDGE_PORT_DEVICE_3='tap2'

Configure NAT with firewalld

To configure NAT with firewalld assign the bridge interface to the internal zone and the interface with access to the network to the external zone:

firewall-cmd --zone=external --add-interface=eth0
firewall-cmd --zone=internal --add-interface=br1

To enable the virtual machines used by openQA to fully access the external network masquerading needs to be enabled on all involved zones:

firewall-cmd --zone=external --add-masquerade
firewall-cmd --zone=internal --add-masquerade

IP forwarding is enabled automatically if masquerading is enabled:

grep 1 /proc/sys/net/ipv4/ip_forward
1

In case the interface is in a trusted network it is possible to accept connections by default by changing the zone target:

firewall-cmd --zone=external --set-target=ACCEPT

Alternatively, you can assign the interface to the trusted zone. Make sure to enable masquerading for the trusted zone as well in this case.

If you are happy with the changes make them persistent:

firewall-cmd --runtime-to-permanent

If you do not currently have the firewalld service running, you can instead use the firewall-cmd-offline command for the configuration. In this case start the firewall and enable the service to run on system startup:

systemctl enable --now firewalld

Also, the firewall-config GUI tool for firewalld can be used for configuration.

For older versions of openSUSE/SLE: Configure NAT with SuSEfirewall2

The IP 10.0.2.2 can be also served as a gateway to access the outside network. For this, NAT between br1 and eth0 must be configured with SuSEfirewall2 or iptables:

# /etc/sysconfig/SuSEfirewall2
FW_DEV_INT="br1"
FW_ROUTE="yes"
FW_MASQUERADE="yes"

Start SuSEfirewall2 and enable the service to start on system startup:

systemctl enable --now SuSEfirewall2

Configure OpenQA Worker Instances

  • Allow worker intstances to run multi-machine jobs:

# /etc/openqa/workers.ini
[global]
WORKER_CLASS = qemu_x86_64,tap
Note
 The number of tap devices should correspond to the number of the running worker instances. For example, if you have set up 3 tap devices, the same number of worker instances should be configured.

  • Enable worker instances to be started on system boot:

systemctl enable openqa-worker@1
systemctl enable openqa-worker@2
systemctl enable openqa-worker@3

Grant CAP_NET_ADMIN Capabilities to QEMU

In order to let QEMU create TAP devices on demand it is required to set CAP_NET_ADMIN capability on QEMU binary file:

zypper in libcap-progs
setcap CAP_NET_ADMIN=ep /usr/bin/qemu-system-x86_64

Configure network interfaces

  • Check the configuration for the eth0 interface:

Important
 Ensure, that eth0 interface is configured in /etc/sysconfig/network/ifcfg-eth0. Otherwise, wicked will not be able to bring up the interface on start and the host will loose network connection: 
# /etc/sysconfig/network/ifcfg-eth0
BOOTPROTO='dhcp'
BROADCAST=''
ETHTOOL_OPTIONS=''
IPADDR=''
MTU=''
NAME=''
NETMASK=''
REMOTE_IPADDR=''
STARTMODE='auto'
DHCLIENT_SET_DEFAULT_ROUTE='yes'

  • Pros of wicked over NetworkManager:

    • Proper IPv6 support

    • openvswitch/vlan/bonding/bridge support - wicked can manage your advanced configuration transparently without the need of extra tools

    • Backwards compatible with ifup scripts

  • Check the network service currently being used:

systemctl show -p Id network.service

If the result is different from Id=wicked.service (e.g. NetworkManager.service), stop the network service:

systemctl disable --now network.service

  • Then switch to wicked and start the service:

systemctl enable --force wicked
systemctl start wicked

  • Bring up the br1 interface:

wicked ifup br1

  • Reboot

Note
 It is also possible to switch the network configuration using YaST.

Debugging Open vSwitch Configuration

Boot sequence with wicked (version 0.6.23 and newer):

  1. openvswitch (as above)

  2. wicked - creates the bridge br1 and tap devices, adds tap devices to the bridge,

  3. SuSEfirewall

  4. os-autoinst-openvswitch - installs openflow rules, handles vlan assignment

The configuration and operation can be checked with the following commands:

ovs-vsctl show # shows the bridge br1, the tap devices are assigned to it
ovs-ofctl dump-flows br1 # shows the rules installed by os-autoinst-openvswitch in table=0

When everything is ok and the machines are able to communicate, the ovs-vsctl should show something like the following:

Bridge "br0"
    Port "br0"
        Interface "br0"
            type: internal
    Port "tap0"
        Interface "tap0"
    Port "tap1"
        tag: 1
        Interface "tap1"
    Port "tap2"
        tag: 1
        Interface "tap2"
  ovs_version: "2.11.1"
Note
 Notice the tag numbers are assigned to tap1 and tap2. They should have the same number.
Note
 If the balance of the tap devices is wrong in the workers.ini the tag cannot be assigned and the communication will be broken.

Check the flow of packets over the network:

  • packets from tapX to br1 create additional rules in table=1

  • packets from br1 to tapX increase packet counts in table=1

  • empty output indicates a problem with os-autoinst-openvswitch service

  • zero packet count or missing rules in table=1 indicate problem with tap devices

iptables -L -v

As long as the SUT has access to external network, there should be a non-zero packet count in the forward chain between the br1 and external interface.

GRE Tunnels

By default all multi-machine workers have to be on single physical machine. You can join multiple physical machines and its ovs bridges together by a GRE tunnel.

If the workers with TAP capability are spread across multiple hosts, the network must be connected. See Open vSwitch documentation for details.

Create a gre_tunnel_preup script (change the remote_ip value correspondingly on both hosts):

# /etc/wicked/scripts/gre_tunnel_preup.sh
#!/bin/sh
action="$1"
bridge="$2"
ovs-vsctl --may-exist add-port $bridge gre1 -- set interface gre1 type=gre options:remote_ip=<IP address of other host>

And call it by PRE_UP_SCRIPT="wicked:gre_tunnel_preup.sh" entry:

# /etc/sysconfig/network/ifcfg-br1
<..>
PRE_UP_SCRIPT="wicked:gre_tunnel_preup.sh"

Allow GRE in firewall:

# /etc/sysconfig/SuSEfirewall2
FW_SERVICES_EXT_IP="GRE"
FW_SERVICES_EXT_TCP="1723"
Note
 When using GRE tunnels keep in mind that virtual machines inside the ovs bridges have to use MTU=1458 for their physical interfaces (eth0, eth1). If you are using support_server/setup.pm the MTU will be set automatically to that value on support_server itself and it does MTU advertisement for DHCP clients as well.