As means for improving I/O performance in virtualized technology, a technology called directed I/O that directly allocates a peripheral component interconnect (PCI) device to a virtual machine is installed as a hardware function (see, for example, Japanese Patent Laid-Open No. 2009-259108, and “Intel (R) Virtualization Technology for directed I/O” Architecture Specification Rev. 1.2).
A portion of the memory of a host computer is allocated as virtual machine memory space in a virtual machine. A physical address recognized by a virtual machine (GPA: Guest-Physical Address) is different from a physical address in a host computer (HPA: Host-Physical Address). A conversion between the GPA and the HPA is required when a direct access is conducted from a virtual machine to a PCI device.
Directed I/O uses hardware (directed I/O hardware) to conduct address conversion (DMA remapping) between the GPA and the HPA to conduct a direct access memory (DMA) transfer between the virtual machine and the PCI device. A PCI configuration space that conducts setting of the PCI device, and device interruption transmission and the like are emulated by virtualization software and provided to the virtual machine. As a result, the virtual machine can treat an actual PCI device as if it is directly connected to the virtual machine.
FIG. 1 illustrates a DMA transfer using DMA remapping. In FIG. 1, a host physical address space s1 is a physical address space in a host computer. A guest physical address space s2 is a physical address space of a virtual machine that operates on the same host computer.
An MMIO regional is a memory region allocated in the host physical address space s1 for a memory mapped I/O corresponding to a PCI device v1 connected to the host computer. An MMIO region a2 is a memory region allocated in the guest physical address space s2 for the memory mapped I/O corresponding to the PCI device v1.
When an access (DMA transfer) to a PCI device is conducted on the virtual machine, the access is conducted with respect to the address in the MMIO region a2. The address in the MMIO region a2 is converted to an address in the MMIO regional by DMA remapping. As a result, access to the PCI device v1 can be achieved.
In another virtualization technology method, migration of a virtual machine operating in one host computer to another host computer while the virtual machine is in an operating state is called “live migration.” Live migration is used, for example, for distributing operational management loads and for mitigating failures and the like. In live migration, the CPU and hardware registers states, memory contents, I/O transfers states and the like all need to match before and after migrating the virtual machine.
A PCI device used in the migration source host computer cannot be used all the time by the virtual machine in the migration target host computer. This is because the PCI device used in the migration source host computer is not connected to the migration target host computer. Therefore, migration of a virtual machine using a PCI device through directed I/O cannot be carried out by live migration.
However, by using a PCI switch, connections from multiple host computers to one PCI device can be shared. Here, “shared” does not necessarily indicate they can be used at the same time. “Shared” indicates that the PCI device connection target host computer can be changed by switching connection relationships in the PCI switch without changing physical wiring. Sharing of a PCI device using a PCI switch has been formulated in the Multi Root I/O Virtualization (MR-IOV) standard using PCI-SIG.
It is possible to share the same PCI device from the virtual machine migration source and migration target host computers using a PCI switch. Therefore, it is conceivable that live migration of a virtual machine using directed I/O is possible between host computers connected to a PCI switch.
However, operations using directed I/O cannot be guaranteed for all PCI devices. For example, it has been found that directed I/O does not function normally with some host adaptors or graphics cards and the like. Conversion between the GPA and HPA during DMA transfers using directed I/O is conducted by hardware. When processing to invalidate this type of conversion is conducted by the hardware, the PCI device may not operate normally. For example, a device driver of the PCI device may try to access a PCI device address (HPA) obtained from the firmware on the PCI device from the virtual machine. In this case, the DMA address conversion may not work and the PCI device may not operate as normal.
For this type of PCI device, it is necessary to build a PCI configuration space to allow access to the correct address even when DMA address conversion does not work. Thus, by matching the MMIO region physical address of the PCI device host computer (HPA) and the MMIO region physical address of the virtual machine (GPA), it has been found that normal operation is possible even when using directed I/O.
FIG. 2 illustrates an HPA and GPA matching state. The same reference numerals used in FIG. 1 are used for the same components in FIG. 2 and explanations thereof will be omitted.
In FIG. 2, the position of the MMIO regional of the host physical address space s1 matches the position of the MMIO region a2 of the guest physical address space s2. In FIG. 2, the matching of the positions indicates that the two MMIO region addresses match.
When conducting live migration of a virtual machine using a PCI device in a state where the HPA and GPA match, it is conceivable that the virtual machine can even use the PCI device in the migration target as long as the state is maintained in the migration target.
However, the firmware (BIOS (basic input/output system)) and the operating system (OS) of the host computer decide the allocation of the physical address of the MMIO region of the PCI device on the host computer at the time of configuration. The physical address of the MMIO region of the PCI device is a value that changes according to the type of host computer and the device configuration of the host computer and the like. Even when a PCI device can be referred to by multiple host computers through the use of a PCI switch, the MMIO region physical addresses of the PCI device on each of the host computers do not necessarily match. That is, the PCI device MMIO region is set to an open region in the migration target host computer that detects a PCI device hot plug from the PCI switch. There is no guarantee that the host physical address of the applicable MMIO region will match the MMIO region host physical address allocated to the PCI device in the migration source host.