XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
•
0 likes•531 views
PCI is a local computer bus for attaching hardware devices in a computer, and is the main peripheral bus on modern x86 systems. As such, having a proper way to emulate it is crucial for Xen to be able to expose both fully emulated devices or passthrough devices to guests. This talk will focus on the current status of PCI emulation in Xen, how and where it is used, what are its main limitations and future plans to improve it in order to be more robust and modular.
Report
Share
More Related Content
Similar to XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
Similar to XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D (20)
More from The Linux Foundation
More from The Linux Foundation (20)
Recently uploaded
Recently uploaded (20)
XPDDS19: Status of PCI Emulation in Xen - Roger Pau Monné, Citrix Systems R&D
- 1. Status of PCI emulation in Xen Roger Pau Monn´e roger.pau@citrix.com Chicago – July 9th, 2019
- 2. PCI bus PCI-passthroughon Xen Moving forward PCI bus Allows attaching hardware devices in a computer. First specification developed by Intel in 1992. Superseded VESA, MCA, EISA, NuBus... Two standards: PCI local bus. PCI Express. Chicago – July 9th, 2019 Status of PCI emulation in Xen 2 / 20
- 3. PCI bus PCI-passthroughon Xen Moving forward PCI slots on a motherboard Obtained from wikipedia author snickerdo. Chicago – July 9th, 2019 Status of PCI emulation in Xen 3 / 20
- 4. PCI bus PCI-passthroughon Xen Moving forward PCI card Chicago – July 9th, 2019 Status of PCI emulation in Xen 4 / 20
- 5. PCI bus PCI-passthroughon Xen Moving forward PCI configuration space The PCI configuration space provides 256bytes or 4096bytes of configuration space to each device. Devices are identified by a 8bit bus, 5bit device and 3bit function integers. First 64bytes is standardized, the rest is device dependent (contains capability structures). Allows for easy discovery of devices. OS can scan the whole bus in order to detect present devices. On x86 can be accessed from IO space (legacy) or memory (enhanced). Chicago – July 9th, 2019 Status of PCI emulation in Xen 5 / 20
- 6. PCI bus PCI-passthroughon Xen Moving forward Legacy PCI configuration access Indirect access using the IO address space. Address port at 0xcf8: 012781011151623243031 E RSV Bus Device Func Register RSV Data port at 0xcfc. Chicago – July 9th, 2019 Status of PCI emulation in Xen 6 / 20
- 7. PCI bus PCI-passthroughon Xen Moving forward Enhanced PCI configuration access Maps the full config space registers into memory space. 0111214151920272831 Base Bus Device Func Register If on 64bit mode bits from 63-32 also contain the base address. Chicago – July 9th, 2019 Status of PCI emulation in Xen 7 / 20
- 8. PCI bus PCI-passthroughon Xen Moving forward PCI header 0781516232431 Status Command 04h Base Address 0 16h Base Address 1 20h Base Address 2 24h Base Address 3 28h Base Address 4 32h Base Address 5 36h CardBus CIS Pointer 40h Subsystem ID Subsystem Vendor ID 44h Expansion ROM Base Address 44h Chicago – July 9th, 2019 Status of PCI emulation in Xen 8 / 20
- 9. PCI bus PCI-passthroughon Xen Moving forward MSI capability 0781516232431 Message control Next pointer Capability ID 00h Message Address [31, 0] 04h Message Address [63, 32] 08h Reserved Message Data 12h Mask Bits 16h Pending bits 20h Chicago – July 9th, 2019 Status of PCI emulation in Xen 9 / 20
- 10. PCI bus PCI-passthroughon Xen Moving forward MSI-X capability 0781516232431 Message control Next pointer Capability ID 00h MSI-X Table Offset BIR 04h PBA Offset BIR 08h 0 31 32 63 64 Vector Control Message Data Upper Address Lower Address Entry 0 ... ... Vector Control Message Data Upper Address Lower Address Entry N Chicago – July 9th, 2019 Status of PCI emulation in Xen 10 / 20
- 11. PCI bus PCI-passthroughon Xen Moving forward PCI handling in Xen PV privileged domain (dom0) gets almost unlimited access to the PCI config space: Xen controls the MSI(-X) mask bits in order to keep a coherent state when doing PCI-passthrough to HVM guests. Read only access is allowed to the MSI-X table and the MSI data and address registers. Passthrough of PCI devices to unprivileged guests: PV guests can access the PCI config space using a Xen PV specific protocol (pciif). HVM guests can access the PCI config space emulated by a device model (QEMU). PVH guests have no PCI-passthrough support yet. Chicago – July 9th, 2019 Status of PCI emulation in Xen 11 / 20
- 12. PCI bus PCI-passthroughon Xen Moving forward PCI-passthrough for domUs Hardware Xen Control Domain (VM0) PV1 HVM1 user-space kernel QEMU evtchn devpciback Chicago – July 9th, 2019 Status of PCI emulation in Xen 12 / 20
- 13. PCI bus PCI-passthroughon Xen Moving forward PCI-passthrough for domUs PV domU communicates directly with pciback using a shared memory ring and a Xen specific protocol. Passthrough to HVM domUs is handled by QEMU, much like emulated devices: PCI config space accesses are forwarded by Xen to QEMU using ioreqs. QEMU emulates or forwards those accesses to the underlying device. Xen directly handles guest writes to the MSI-X mask bits for performance reasons. Device MMIO regions (BARs) are directly mapped to the guest physmap, except for the MSI-X region if present. Chicago – July 9th, 2019 Status of PCI emulation in Xen 13 / 20
- 14. PCI bus PCI-passthroughon Xen Moving forward PV dom0 Has almost unlimited read/write access to the configuration space except for certain parts of the MSI(-X) capabilities. Has to use hypercalls to deal with certain capabilities: MSI/MSI-X. Is fully trusted to not misbehave. Chicago – July 9th, 2019 Status of PCI emulation in Xen 14 / 20
- 15. PCI bus PCI-passthroughon Xen Moving forward PVH dom0 PVH is a HVM guest from Xen’s point of view. HVM-like access to the configuration space: Transparent access to the MSI/MSI-X capabilities. Transparent mapping of BARs into the physmap and handling of writes to the BAR registers. Chicago – July 9th, 2019 Status of PCI emulation in Xen 15 / 20
- 16. PCI bus PCI-passthroughon Xen Moving forward PVH dom0 Current PCI-passthrough code for HVM is in QEMU. Impossible to use QEMU for PVH dom0. No re-use of the QEMU PCI-passthrough code: would need heavy modifications that would make sharing changes very difficult. Added a PCI config space mediator to the hypervisor: vPCI. Chicago – July 9th, 2019 Status of PCI emulation in Xen 16 / 20
- 17. PCI bus PCI-passthroughon Xen Moving forward PCI-passthrough mediators in Xen QEMU (user-space) and Xen for MSI-X mask bits for HVM domUs. pciback (hardware domain OS) for PV domUs. Direct access / hypercalls for PV dom0. vPCI (hypervisor) for PVH dom0. Chicago – July 9th, 2019 Status of PCI emulation in Xen 17 / 20
- 18. PCI bus PCI-passthroughon Xen Moving forward Shortcomings No support for VFIO/MDEV on Xen: Threatening support for vGPU/XenGT in future releases. 3 different code bases to deal with PCI config space accesses: More maintainership work. Non uniform behaviour across different guests types. Chicago – July 9th, 2019 Status of PCI emulation in Xen 18 / 20
- 19. PCI bus PCI-passthroughon Xen Moving forward Future items Re-work vPCI so it can be used both inside the hypervisor and in user-space. Could be used by HVM and PVH guests as a standalone PCI-passthrough utility. Unify PCI-passthrough for HVM and PVH both domU and dom0 into a single code-base. Add support for the extended config space to HVM domUs: allow to passthrough PCIe capabilities. Add support to passthrough SRIOV capability to vPCI, for PVH dom0. Chicago – July 9th, 2019 Status of PCI emulation in Xen 19 / 20
- 20. PCI bus PCI-passthroughon Xen Moving forward Q&A Thanks Questions? Chicago – July 9th, 2019 Status of PCI emulation in Xen 20 / 20