In many computer systems, peripheral devices communicate with the central processing unit (CPU) and with one another over a peripheral component bus, such as the PCI-Express® (PCIe®) bus. Such peripheral devices may include, for example, a solid state drive (SSD), a network interface controller (NIC), and various accelerator modules, such as a graphics processing unit (GPU).
Methods for directly accessing the local memory of a peripheral device via PCIe and other peripheral component buses are known in the art. For example, U.S. Patent Application Publication 2015/0347349, which is assigned to the assignee of the present patent application and whose disclosure is incorporated herein by reference, describes a method for communicating between at least first and second devices over a bus in accordance with a bus address space, including providing direct access over the bus to a local address space of the first device by mapping at least some of the addresses of the local address space to the bus address space. The term “direct access” means that data can be transferred between devices, over the bus, with no involvement of the software running on the CPU in the data plane.
As another example, GPUDirect RDMA is an application program interface (API) that supports interaction between an InfiniBand™ NIC (referred to as a host channel adapter, or HCA) and peer memory clients, such as GPUs. It is distributed by Mellanox Technologies Ltd. (Yokneam, Israel). This API provides a direct P2P (peer-to-peer) data path between the GPU memory and Mellanox HCA devices. It enables the HCA to read and write peer memory data buffers, and thus allows RDMA-based applications to use the computing power of the peer device without the need to copy data to host memory.
Some PCIe bus components support Access Control Services (ACS), as defined in section 6.12 (pages 533-542) of the PCI Express Base Specification (Rev. 3.0, referred to hereinbelow simply as the “PCIe specification”). ACS provides a number of facilities, including “ACS Source Validation” based on a “Bus Number” field in the Requester ID of transactions transmitted over the bus. When ACS Source Validation is enabled, the downstream ports of PCIe switches test the bus number of each upstream request received by the port to determine whether it is within a certain specified bus number “aperture” of the port. If this bus number is not within the prescribed aperture, an ACS Violation error is reported.
As another alternative, “ACS P2P Request Redirect” can be used to cause PCIe switches to redirect peer-to-peer requests that they receive to the root complex for request validation. The root complex then determines whether the request should be “reflected” back downstream toward its original target, or blocked as an ACS Violation error.
In a similar vein, U.S. Patent Application Publication 2006/0179195 describes a method and an apparatus for restricting input/output (I/O) device peer-to-peer (P2P) operations in a data processing system, in a manner that is said to improve reliability, availability, and serviceability. P2P control logic performs a lookup of P2P lookup table entries. Each P2P lookup table entry comprises bus, device and function number fields, optional control fields, and an accept/reject bit. Upon receiving a communication request from a requesting I/O device, the P2P control logic implemented in either a logical bridge or an I/O device identifies the requester ID of the request and determines whether a match exists in the P2P lookup table entries. If a match is found and the accept/reject bit is enabled, I/O operations can be received from the requester.