Enabling SRIOV on RHEL 7

Prerequisites:

SRIOV capable hardware
System BIOS configured with SRIOV enabled
System installed with RHEL 7.2 with sufficient disk space allocated to /

Once the system is ready with RHEL 7.2 and SRIOV feature enabled from BIOS, make the following changes to the system configuration to enable Virtual Function (SRIOV) on the NIC card:

Add “ intel_iommu=on igb.max_vfs=7" to the default GRUB file.
[root@rhops2 ~]# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rhgb quiet intel_iommu=on igb.max_vfs=7" GRUB_DISABLE_RECOVERY="true"
GRUB_DISABLE_RECOVERY="true"
Re-compile GRUB to make this change the default on bootup. [root@rhops2 ~]# /sbin/grub2-mkconfig -o /boot/grub2/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-327.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-327.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-156e082173f7431fa38c0afa16c3704a
Found initrd image: /boot/initramfs-0-rescue-156e082173f7431fa38c0afa16c3704a.img
done
Create or edit network the driver config file to enable Virtual functions on the NIC card. Specify the number of required vifs on each network interface. [This is OPTIONAL if GRUB is already modified as mentioned in step #1.]
[root@rhosp ~]# cat /etc/modprobe.d/igb.conf
options igb max_vfs=7
Reboot the system to reflect performance changes on the system.
[root@rhosp ~]# reboot
Upon system reboot, Virtual functions will be enabled on network interfaces.
[root@rhosp ~]# lspci --nn | grep -i net

Setting up OpenStack with SRIOV

Modify the OpenStack config files to enable SRIOV support.
1. /etc/nova/nova.conf
2. /etc/neutron/plugins/ml2/ml2_conf.ini
3. /etc/neutron/plugins/ml2/ml2_conf_sriov.ini
4. /usr/lib/systemd/system/neutron-server.service
Find out the PCI address for the Virtual functions that you intend to use and add it to /etc/nova/nova.conf.
[root@rhops2 ~]# lspci -nn | grep -i net
01:00.0 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection
[8086:1521] (rev 01)
01:00.1 Ethernet controller [0200]: Intel Corporation I350 Gigabit Network Connection
[8086:1521] (rev 01)
01:10.0 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:10.1 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:10.4 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:10.5 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:11.0 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:11.1 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:11.4 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:11.5 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:12.0 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:12.1 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:12.4 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:12.5 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:13.0 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
01:13.1 Ethernet controller [0200]: Intel Corporation I350 Ethernet Controller Virtual
Function [8086:1520] (rev 01)
Since this server has two interfaces, you will be using VFs on one interface for “MGMT” and the VFs on other interface for “LAN”, “HA” and “LAN2.” If the server has more than two interfaces, they can be mapped differently. (Infoblox recommends that you map the NIOS interfaces individually with virtual functions bound to a different physical interface. This helps HA failover situations).
[root@rhops2 ~]# cat /etc/nova/nova.conf
……
……
……
# White list of PCI devices available to VMs. For example:
# pci_passthrough_whitelist = [{"vendor_id": "8086",
# "product_id": "0443"}] (multi valued)
#pci_passthrough_whitelist=
## Virtual Functions on eth0
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:10.0",
"physical_network":"mgmt"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:10.1",
"physical_network":"lan1"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:10.4",
"physical_network":"mgmt"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:10.5",
"physical_network":"ha"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:11.0",
"physical_network":"mgmt"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:11.1",
"physical_network":"lan2"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:11.4",
"physical_network":"mgmt"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:11.5",
"physical_network":"lan1"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:12.0",
"physical_network":"mgmt"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:12.1",
"physical_network":"ha"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:12.4",
"physical_network":"mgmt"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:12.5",
"physical_network":"lan2"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:13.0",
"physical_network":"mgmt"}
pci_passthrough_whitelist = {"vendor_id": "8086","product_id": "1520", "address":"01:13.1",
"physical_network":"lan1"}
……
……
……
#
# Options defined in nova.scheduler.host_manager
#

# Filter classes available to the scheduler which may be
# specified more than once. An entry of
# "nova.scheduler.filters.all_filters" maps to all filters
# included with nova. (multi valued)
scheduler_available_filters=nova.scheduler.filters.all_filters

# Which filter class names to use for filtering hosts when not
# specified in the request. (list value)
#scheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,Server
GroupAntiAffinityFilter,ServerGroupAffinityFilterscheduler_default_filters=RetryFilter,AvailabilityZoneFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,CoreFilter,PciPassthroughFilter
……
……
……

4. Now add the sriovnic option and flat_network configuration to the ml2_conf.ini file.

[root@rhops2 ~]# cat /etc/neutron/plugins/ml2/ml2_conf.ini | grep -v ^$ | grep -v ^#
[ml2]
type_drivers = flat,vlan,gre,vxlan
tenant_network_types = vxlan
mechanism_drivers =openvswitch,sriovnicswitch
[ml2_type_flat]
flat_networks = mgmt,lan1,ha,lan2
[ml2_type_vlan]
[ml2_type_gre]
[ml2_type_vxlan]
vni_ranges =10:100
vxlan_group =224.0.0.1
[securitygroup]
enable_security_group = True
firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver
[ovs]
local_ip = 10.36.31.90

5. Add supported PCI “vendor id : product id” and physical device mapping to the /etc/neutron/plugins/ml2/ml2_conf_sriov.ini file.

[root@rhops2 ~]# cat /etc/neutron/plugins/ml2/ml2_conf_sriov.ini | grep -v ^$ | grep -v ^#
[ml2_sriov]
supported_pci_vendor_devs = 8086:1520
agent_required = False
[sriov_nic]
physical_device_mappings = mgmt:eno1,lan1:eno2,ha:eno2,lan2:eno2

Add “ml2_conf_sriov.ini” config file details in
“/usr/lib/systemd/system/neutron-server.service”

[root@rhops2 ~]# cat /usr/lib/systemd/system/neutron-server.service
[Unit]
Description=OpenStack Neutron Server
After=syslog.target network.target

[Service]
Type=notify
User=neutron
ExecStart=/usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf
--config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf
--config-file /etc/neutron/plugin.ini --config-dir /etc/neutron/conf.d/common --config-dir
/etc/neutron/conf.d/neutron-server --config-file
/etc/neutron/plugins/ml2/ml2_conf_sriov.ini --log-file /var/log/neutron/server.log
PrivateTmp=true
NotifyAccess=all
KillMode=process
[Install]
WantedBy=multi-user.target

6. Now restart the OpenStack service for the new configuration to take affect.

[root@rhops2 ~]# openstack-service restart
Warning: neutron-server.service changed on disk. Run 'systemctl daemon-reload' to reload units.

7. Now create networks in OpenStack setup and map interface (sriov) to these new networks.

[root@rhops2 ~(keystone_admin)]# neutron net-list
+--------------------------------------+---------+------------------------------------------------------+
| id | name | subnets |
+--------------------------------------+---------+------------------------------------------------------+
| 43a84db7-acf6-4404-b6af-f8601fbb8eec | public | 4d176229-3207-4e5e-b1eb-b638accf59f5 172.24.4.224/28 |
| ec7e97b9-8f2f-41c4-bac8-c91352fb2f2a | private | f4c9ded8-07aa-42da-b962-81fb9e691403 10.0.0.0/24 |
+--------------------------------------+---------+------------------------------------------------------+

Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up    | True |
| id    | 37cae5b8-4598-43e6-8c53-fd23f2c7c45c |
| mtu | 0    |
| name                      | lan1                                 |
| provider:network_type | flat                                 |
| provider:physical_network | lan1                                 |
| provider:segmentation_id |    |
| router:external           | False    |
| shared    | False    |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | d57f8170a21a4f5f970fb7a72f3202a6 |
+---------------------------+--------------------------------------+
[root@rhops2 ~(keystone_admin)]# neutron net-create --provider:physical_network=ha
--provider:network_type=flat ha

Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id    | 0e7efa05-82b9-4498-9915-48a1748a0238 |
| mtu | 0                                    |
| name    | ha                                   |
| provider:network_type     | flat                                 |
| provider:physical_network | ha |
| provider:segmentation_id |                                      |
| router:external           | False    |
| shared    | False    |
| status                    | ACTIVE                               |
| subnets |    |
| tenant_id | d57f8170a21a4f5f970fb7a72f3202a6     |
+---------------------------+--------------------------------------+
[root@rhops2 ~(keystone_admin)]# neutron net-create --provider:physical_network=lan2
--provider:network_type=flat lan2
Created a new network:
+---------------------------+--------------------------------------+
| Field | Value    |
+---------------------------+--------------------------------------+
| admin_state_up    | True                                 |
| id    | 073b6886-7875-47af-b7aa-44b792099337 |
| mtu                       | 0                                    |
| name                      | lan2 |
| provider:network_type     | flat                                 |
| provider:physical_network | lan2                                 |
| provider:segmentation_id |                                      |
| router:external | False    |
| shared    | False    |
| status    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | d57f8170a21a4f5f970fb7a72f3202a6     |
+---------------------------+--------------------------------------+
[root@rhops2 ~(keystone_admin)]# neutron subnet-create --name subnet-mgmt mgmt 10.36.0.0/16
--allocation-pool start=10.36.31.221,end=10.36.31.225 --disable-dhcp
Created a new subnet:
+-------------------+--------------------------------------------------+
| Field | Value                                            |
+-------------------+--------------------------------------------------+
| allocation_pools | {"start": "10.36.31.221", "end": "10.36.31.225"} |
| cidr              | 10.36.0.0/16 |
| dns_nameservers   |                                                  |
| enable_dhcp       | False    |
| gateway_ip        | 10.36.0.1    |
| host_routes |                                                  |
| id                | 824f9edb-3b30-4179-88f1-eb24a8a1b53b             |
| ip_version    | 4                                                |
| ipv6_address_mode |                                                  |
| ipv6_ra_mode      |                                                  |
| name              | subnet-mgmt    |
| network_id        | a72836cb-87b5-46c8-ada9-efbde7ed1698 |
| subnetpool_id     |    |
| tenant_id | d57f8170a21a4f5f970fb7a72f3202a6 |
+-------------------+--------------------------------------------------+
[root@rhops2 ~(keystone_admin)]# neutron subnet-create --name subnet-lan1 lan1
10.34.31.0/24 --allocation-pool start=10.34.31.221,end=10.34.31.225 --disable-dhcp
Created a new subnet:
+-------------------+--------------------------------------------------+
| Field             | Value                                            |
+-------------------+--------------------------------------------------+
| allocation_pools | {"start": "10.34.31.221", "end": "10.34.31.225"} |
| cidr              | 10.34.31.0/24                                    |
| dns_nameservers   |                                                  |
| enable_dhcp       | False                                            |
| gateway_ip        | 10.34.31.1 |
| host_routes       |                                                  |
| id                | fefb9dda-d6e6-43b6-af59-db6f747a7440 |
| ip_version        | 4                                                |
| ipv6_address_mode |                                                  |
| ipv6_ra_mode    |    |
| name              | subnet-lan1                                      |
| network_id    | 37cae5b8-4598-43e6-8c53-fd23f2c7c45c             |

| subnetpool_id | |
| tenant_id | d57f8170a21a4f5f970fb7a72f3202a6 |
+-------------------+--------------------------------------------------+

[root@rhops2 ~(keystone_admin)]# neutron subnet-create --name subnet-ha ha 10.34.31.0/24
--allocation-pool start=10.34.31.226,end=10.34.31.230 --disable-dhcp
Created a new subnet:
+-------------------+--------------------------------------------------+
| Field             | Value                                            |
+-------------------+--------------------------------------------------+
| allocation_pools | {"start": "10.34.31.226", "end": "10.34.31.230"} |
| cidr              | 10.34.31.0/24    |
| dns_nameservers   |                                                  |
| enable_dhcp       | False    |
| gateway_ip        | 10.34.31.1                                       |
| host_routes       |                                                  |
| id                | 2a81d4fc-2790-4072-9e83-ac075fcfbdde             |
| ip_version    | 4                                                |
| ipv6_address_mode |                                                  |
| ipv6_ra_mode    |                                                  |
| name              | subnet-ha    |
| network_id    | 0e7efa05-82b9-4498-9915-48a1748a0238 |
| subnetpool_id |                                                  |
| tenant_id | d57f8170a21a4f5f970fb7a72f3202a6                 |
+-------------------+--------------------------------------------------+
[root@rhops2 ~(keystone_admin)]# neutron subnet-create --name subnet-lan2 lan2
10.34.31.0/24 --allocation-pool start=10.34.31.231,end=10.34.31.235 --disable-dhcp
Created a new subnet:
+-------------------+--------------------------------------------------+
| Field | Value    |
+-------------------+--------------------------------------------------+
| allocation_pools | {"start": "10.34.31.231", "end": "10.34.31.235"} |
| cidr              | 10.34.31.0/24                                    |
| dns_nameservers   |    |
| enable_dhcp | False                                            |
| gateway_ip    | 10.34.31.1                                       |
| host_routes |                                                  |
| id                | 485d714e-f34c-4994-9bff-ef9edecf507e             |
| ip_version        | 4    |
| ipv6_address_mode |                                                  |
| ipv6_ra_mode      |    |
| name              | subnet-lan2    |
| network_id    | 073b6886-7875-47af-b7aa-44b792099337             |
| subnetpool_id |    |
| tenant_id         | d57f8170a21a4f5f970fb7a72f3202a6                 |
+-------------------+--------------------------------------------------+
[root@rhops2 ~(keystone_admin)]# neutron net-list
+--------------------------------------+---------+------------------------------------------------------+
| id                                   | name    | subnets                                              |
+--------------------------------------+---------+------------------------------------------------------+
| 073b6886-7875-47af-b7aa-44b792099337 | lan2    | 485d714e-f34c-4994-9bff-ef9edecf507e 10.34.31.0/24   |
| 0e7efa05-82b9-4498-9915-48a1748a0238 | ha      | 2a81d4fc-2790-4072-9e83-ac075fcfbdde 10.34.31.0/24   |
| 37cae5b8-4598-43e6-8c53-fd23f2c7c45c | lan1    | fefb9dda-d6e6-43b6-af59-db6f747a7440 10.34.31.0/24 |
| 43a84db7-acf6-4404-b6af-f8601fbb8eec | public | 4d176229-3207-4e5e-b1eb-b638accf59f5 172.24.4.224/28 |
| a72836cb-87b5-46c8-ada9-efbde7ed1698 | mgmt    | 824f9edb-3b30-4179-88f1-eb24a8a1b53b 10.36.0.0/16    |
| ec7e97b9-8f2f-41c4-bac8-c91352fb2f2a | private | f4c9ded8-07aa-42da-b962-81fb9e691403 10.0.0.0/24     |
+--------------------------------------+---------+------------------------------------------------------+

Infoblox Installation Guide vNIOS™ for OpenStack with SRIOV

Enabling SRIOV on RHEL 7

Analytics

Prerequisites:

Related content