Testing VFIO_PLATFORM with the PL330 DMA Controller

VFIO Test Cases Introduction

Modern hardware components include Direct Memory Access (DMA) capabilities, which allow them to quickly deliver data to the running operating system. DMA accesses are problematic when mapping a device either to a user space application or to a guest virtual machine. Not only the hardware will be configured using virtual (or intermediate physical) memory addresses, other security concerns also apply since the guest operating system cannot be trusted with direct access to the host’s or other VMs memory space.

An IOMMU is a hardware block, similar to a MMU of CPU which intercepts any memory access from programmable or not hardware accelerators, and translates the memory addresses, from virtual addresses to physical addresses. Hardware components behind a properly configured IOMMU will operate under the illusion that they are directly accessing the physical memory of the machine, while they have actually been mapped to the memory space of a virtual machine. Additionally, the IOMMU will prevent the associated devices from accessing memory beyond the mappings that have been assigned to them in the page tables.

The VFIO framework is featured in the Linux kernel in order to fulfill a precise requirement, which is the possibility to realize user space device drivers, exploiting the capabilities of a DMA capable device behind an IOMMU. This further applies to Virtual Machines, driven by QEMU/KVM, as the capability to assign hardware devices directly to a guest OS, while preserving the isolation properties of virtualization and the security of the system.

Virtual Open Systems, is actively developing the VFIO_PLATFORM driver, which allows to use VFIO on ARM based systems. With this driver, devices on an ARM based SoC can be bound to VFIO, and be used via the VFIO API. This can be used with KVM on ARM virtual machines for device assignment to dramatically improve I/O performance. The goal of this document is to guide the reader, step by step, not only on building the VFIO_PLATFORM driver on ARM, but also through two different test cases that Virtual Open Systems has implemented to validate VFIO on ARM:

The first will run through all the steps necessary to compile and configure an user space test case and VFIO driver example, based on the DMA-330 DMA Controller (also known as PL330 DMA controller). This device is an AMBA peripheral developed by ARM; it is easily obtainable using FastModels, building the corresponding model available among all the examples included in the portfolio. As a DMAC, its purpose is to copy data memory-to-memory, memory-to-device or device-to-memory. However, in order to validate the VFIO implementation on ARM, the VFIO test case we are going to use will instruct the device to only make memory-to-memory data transfers.

The second use-case shows how we can configure QEMU to directly expose a real device to a virtual machine. The QEMU upstream version does not yet allow binding of platform devices to virtual machines; this feature is reserved for PCI devices (PCI passthrough) only. However, the community is now moving towards integrating the VFIO API for platform devices into QEMU, in order to guarantee the passthrough feature for this type of devices as well. Virtual Open Systems is contributing towards VFIO and we present the latest state of VFIO integration in QEMU, at the time of writing of this guide. This use-case illustrates the steps required to bind the DMA-330 controller to a virtual machine: in this scenario QEMU will mostly do all of the job for us, requiring only some basic information for locating the device to be bound. The guest system will not be aware of the underlying mechanism and will probe the device seamlessly with an untouched kernel driver.

Setting up an ARM Linux Host with VFIO_PLATFORM

In this section we are going to prepare a Linux distribution which will be used through the rest of the guide. We will set up all the required elements:

install the required packages for cross compiling

compile the model which includes the PL330 DMA Controller and an ARM SMMU-400

set up a NFS resource to accommodate the user space file system

compile the host Linux kernel

configure the final details to boot the model

These first steps are in common with the two alternative uses of VFIO that we feature in this guide. After this chapter, we will look into the necessary steps to build the two test cases.

$ sudo apt-get update && sudo apt-get install build-essential git libncurses5-dev

$ sudo apt-get install gcc-arm-linux-gnueabi coreutils-armel-cross libc6-armel-cross \
        libc6-dev-armel-cross linux-libc-dev-armel-cross binutils-arm-linux-gnueabi  \
        debootstrap qemu-user-static

$ sudo apt-get install nfs-kernel-server nfs-common

$ sudo mkdir /srv/vfio_host

/srv/vfio_host 192.168.0.0/255.255.0.0(rw,sync,no_root_squash,no_subtree_check,insecure)

$ cd /srv/
$ sudo qemu-debootstrap --arch=armel quantal ./vfio_host

###### Ubuntu Main Repos
deb http://ports.ubuntu.com/ quantal main restricted universe

###### Ubuntu Update Repos
deb http://ports.ununtu.com/ quantal-security main restricted universe

$ sudo cp /srv/vfio_host/etc/init/tty1.conf /srv/vfio_host/etc/init/ttyAMA0.conf

exec /sbin/getty -8 38400 ttyAMA0

$ git clone git://github.com/virtualopensystems/linux-kvm-arm.git
$ cd linux-kvm-arm
$ git checkout vfio-platform-v6-guide
$ CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm make menuconfig

$ cp arch/arm/configs/vexpress_defconfig .config
$ scripts/config -e CONFIG_ARM_LPAE
$ scripts/config -e CONFIG_PROC_DEVICETREE
$ scripts/config -e CONFIG_EXT3_FS
$ scripts/config -e CONFIG_VIRTUALIZATION
$ scripts/config -e CONFIG_KVM
$ scripts/config -e CONFIG_KVM_ARM_TIMER
$ scripts/config -e CONFIG_ARM_SMMU
$ scripts/config -e CONFIG_VFIO
$ scripts/config -e CONFIG_VFIO_PLATFORM
$ scripts/config -e CONFIG_KVM_ARM_VGIC
$ scripts/config --set-val CONFIG_KVM_ARM_MAX_VCPUS 4
$ scripts/config --set-val CONFIG_ARM_DMA_IOMMU_ALIGNMENT 8
$ scripts/config -e CONFIG_EXT3_DEFAULTS_TO_ORDERED
$ scripts/config -e CONFIG_EXT3_FS_XATTR
$ scripts/config -e CONFIG_EXT3_FS_POSIX_ACL
$ scripts/config -e CONFIG_EXT3_FS_SECURITY
$ scripts/config -e CONFIG_VIRTIO
$ scripts/config -e CONFIG_VIRTIO_BLK
$ scripts/config -e CONFIG_VIRTIO_RING
$ scripts/config -e CONFIG_VIRTIO_MMIO
$ scripts/config -e CONFIG_DMA_ENGINE
$ scripts/config -e CONFIG_DMADEVICES
$ scripts/config -e CONFIG_PL330_DMA
$ scripts/config -e CONFIG_VOSYS_DMATEST

$ CROSS_COMPILE=arm-linux-gnueabi- ARCH=arm make zImage

$ scripts/dtc/dtc -I dts -O dtb -o /path/to/tree.dtb arch/arm/boot/dts/rtsm_ve-cortex_a15x4_a7x4.dts

#!/bin/bash

MODEL_PATH='<PATH_TO_MODEL>'

$MODEL_PATH/isim_system \
-a coretile.cluster0.*=boot-wrapper/linux-system-semi.axf \
-a coretile.cluster1.*=boot-wrapper/linux-system-semi.axf \
-C motherboard.smsc_91c111.enabled=1 \
-C motherboard.hostbridge.interfaceName="tap0" \
-C coretile.cluster0.cpu0.semihosting-enable=1 \
-C coretile.dmac0_int_base=64 \
-C coretile.dmac1_int_base=96 \
-C coretile.dmac2_int_base=128 \
-C coretile.dmac3_int_base=168 \
-C coretile.mmu400_dma330.dmac0.p_irq_nsecure="true" \
-C coretile.mmu400_dma330.dmac1.p_irq_nsecure="true" \
-C coretile.mmu400_dma330.dmac2.p_irq_nsecure="true" \
-C coretile.mmu400_dma330.dmac3.p_irq_nsecure="true" \
-C coretile.cluster0.cpu0.semihosting-cmd_line="\
--kernel <PATH_TO_ZIMAGE> \
--dtb tree.dtb -- earlyprintk console=ttyAMA0 mem=2048M \
root=/dev/nfs nfsroot=<YOUR_IP>:/srv/vfio_host/ rw ip=dhcp"

#!/bin/bash

gateway="192.168.1.1"
bridge_ip="192.168.1.80"

sudo brctl addbr br0
sudo tunctl -u <USERNAME>

sudo ifconfig eth0 0.0.0.0 promisc up
sudo ifconfig tap0 0.0.0.0 promisc up

sudo brctl addif br0 eth0
sudo brctl addif br0 tap0
sudo brctl stp br0 off

sudo ifconfig br0 $bridge_ip netmask 255.255.255.0 broadcast 192.168.1.255
sudo route add default gw $gateway

$ sh boot_model.sh

$ ls /sys/devices/ | grep dma

# echo 1 > /sys/kernel/debug/vosys_dmatest/start
# dmesg |grep "vosys dmatest"

$ ls /sys/module/ | grep vfio_platform

Test Case 1: A VFIO_PLATFORM Userspace Driver

As already mentioned in the introduction, the first VFIO test case should validate the possibility to have a complete user space driver for a device behind the IOMMU. Virtual Open Systems has implemented a reference test case based on the PL330 DMA controller in the FastModels. To clone the code from the Virtual Open Systems repository:

# apt-get install build-essential libglib2.0-dev git
$ git clone https://github.com/virtualopensystems/vfio-host-test.git
$ cd vfio-vhost-test

This is a suite intended to test VFIO for platform devices. The suite relays on a bash script to check the devices available that could be potentially used as target for the VFIO tests. The test will perform some general, yet vital checks and then execute some specific code of the device being tested. At the moment, only the DMA330 has a specific test code available that will instruct the device to perform a simple DMA transfer, from Userspace.

Since we are dealing with the DMA330, we can instruct the test suite to perform directly the device specific code:

# ./vfio_test_device.sh -s

When asked, press the index of the device to test ("0" for 2c0a0000.dma); if everything goes like it should, the output will confirm the successful outcome of the test.

If you want to have more visibility on what the test actually does, we suggest to execute the following commands instead of using the automated script. First of all, we need to compile the DMA330 test:

$ cd src_test/pl330/ && make

Before running the test, we need to configure VFIO to bind with the target device. Since the model provides four different DMA controllers, we need to choose one of them as target. Here we have picked the first one (2c0a0000.dma).

# echo vfio-platform  > /sys/bus/platform/devices/2c0a0000.dma/driver_override
# echo 2c0a0000.dma > /sys/bus/platform/drivers_probe

Now that the device is bound, we need to find the IOMMU group it is associated with. To do that, we may simply run:

$ ls -l /sys/devices/2c0a0000.dma/iommu_groups

and look at the last number of the output (in our case "0"). Depending on this value, a new file will be created in /dev/vfio/ which will be the first parameter of the next command. Finally, we can run the test:

# ./test_pl330_vfio_driver /dev/vfio/0 2c0a0000.dma

If you wish to test another DMA controller, you can change the above commands accordingly. Please note that, in order to test a controller other than the first available, an additional modification in the device tree might be required so that the kernel can see the device as attached to the platform bus. To do that, the value "arm,primecell" has to be removed from the compatible string of the corresponding dma device.

Test Case 2: Device Passthrough on a QEMU/KVM Guest

In this section we will deal with the second, and more interesting, use case of VFIO: an example of device assignment for a QEMU/KVM virtual machine. This time the scenario is a little more complicated; it is formed by an host (we will use the same machine we were using previously) and by a QEMU guest, that we will build soon.

The guest will use the kernel DMA test module to set up a DMA transfer which in turn will use the (unmodified) PL330 guest kernel driver. The execution of the driver code in the guest will not involve any trap to the host for emulation, but will instead instruct directly the real hardware device.