PCIe 5.0: Harnessing the Power of High-Speed Data Transfers

PCIe, or Peripheral Component Interconnect Express, is a high-speed expansion bus that is used to connect a wide variety of devices to a computer’s motherboard. PCIe 5.0 is the fifth generation of the PCIe standard, and it offers significant performance improvements over previous generations. It not only provides a high-speed, reliable data transfer channel but also has the flexibility to scale to meet the demands of processing large amounts of data, for example AI, machine learning, HPC, and data centers.

Advantages of PCIe 5.0

• Increased bandwidth: PCIe 5.0 offers twice the bandwidth of PCIe 4.0. A PCIe 5.0 x16 slot can provide up to 128 GB/s of bandwidth, which is double the bandwidth of a PCIe 4.0 x16 slot. This can significantly improve the performance of devices such as graphics cards and SSDs, and networking adapters.
• Lower latency: PCIe 5.0 has lower latency than previous generations, which means that data can be transferred between the CPU and devices more quickly. This can improve the performance of applications that are sensitive to latency, such as real-time audio and video applications.
• Improved power efficiency: PCIe 5.0 is more power efficient than previous generations, which can help to reduce the overall power consumption of a system.
• Backwards compatibility: PCIe 5.0 is backwards compatible with PCIe 4.0 and PCIe 3.0 devices, so you can still use your existing devices with a PCIe 5.0 system, allowing for easy integration with existing hardware and software.
• Support for CXL: PCIe 5.0 supports the Compute Express Link (CXL) standard, an important new interface that will be widely deployed in datacenters in the coming years.

CXL
Compute Express Link (CXL) is an open industry standard interconnect first unveiled in March 2019, offering high-bandwidth, low-latency connectivity between the host processor and devices including accelerators, memory expansion, and smart I/O devices. It represents an important technology that leverages the dependability and performance of the PCIe physical and electrical interface and introduces new high bandwidth and low latency protocols on top of PCIe for attaching accelerators and memory buffers. CXL has the benefit of supporting both standard PCIe devices as well as CXL devices.

CXL 1.1 is the first productized version of CXL. The CXL Specification 1.1 was released in June, 2019. On November 10, 2020, the CXL Specification 2.0 was released. The new version adds support for CXL switching, to allow connecting multiple CXL 1.x and 2.0 devices to a CXL 2.0 host processor, and/or pooling each device to multiple host processors, in distributed shared memory and disaggregated storage configurations; it also implements device integrity and data encryption. There is no bandwidth increase from CXL 1.x, because CXL 2.0 still utilizes PCIe 5.0 PHY.

On August 2, 2022, the CXL 3.0 was released, based on PCIe 6.0 physical interface and PAM-4 coding with double the bandwidth; new features include fabrics capabilities with multi-level switching and multiple device types per port, and enhanced coherency with peer-to-peer DMA and memory sharing.

CXL is deployed as 3 protocols:

• io: CXL.io is nearly identical to the PCIe transaction layer, and supports basic functionality like device discovery, configuration, initialization, and I/O virtualization. All CXL devices are required to implement CXL.io.
• cache: it enables a device to cache data from the host memory, which allows CXL devices to cache host memory with low latency.
• mem: provides a host processor with direct access to memory attached to a CXL device.

CXL is designed to support three primary device types:

• Type 1 (CXL.io and CXL.cache) – specialized accelerators (such as smart NIC) with no local memory. Devices rely on coherent access to host CPU memory.
• Type 2 (CXL.io, CXL.cache and CXL.mem) – general-purpose accelerators (GPU, ASIC or FPGA) with high-performance GDDR or HBM local memory. Devices can coherently access host CPU’s memory and/or provide coherent or non-coherent access to device local memory from the host CPU.
• Type 3 (CXL.io and CXL.mem) – memory expansion boards and persistent memory. Devices provide host CPU with low-latency access to local DRAM or byte-addressible non-volatile storage.

PCIe 5.0 Supported Devices
Apart from the existing PCIe 4.0/3.0 devices that are backwardly supported, the components which can benefit the most from PCIe 5.0 are most likely high-speed network adapters, NVMe storage, and CXL memory modules.

• For high-end networking like 400Gb Ethernet solutions and dual 200Gb/s InfiniBand, the PCIe 5.0 operates at full duplex transmissions and provides up to 128GB/s in bandwidth.
• At PCIe 4.0 speeds with maximum 8GB/s of constant data transfer, the fastest NVMe SSDs are approaching that 8GB/s limit. A PCIe 5.0 SSD has up to 75% increase of maximum 14GB/s.
• CXL Type 3 devices will enable expansion of memory in a system via the PCIe bus, which increases server performance and reduce total cost of ownership (TCO) through memory expansion and pooling.

There are few PCIe 5.0 GPU and FPGA cards launched on the market now, but they are becoming more widely available.

Conclusion
AEWIN’s latest PCIe Gen 5 based network computing platforms are equipped with 8x total PCIe x8 expansion bays to support up to 8x 200Gbps connections with full throughput capabilities. Beyond networking, PCIe Gen 5 has great potential for accelerated computing. Although bandwidth through the PCIe connection is generally not an issue for GPU accelerated computing, being able to transfer the same amount of data in shorter time will ensure that GPU utilization will be maximized and never waiting for data.

The major advantages of deploying AEWIN PCIe Gen 5 platforms include TCO savings, rack space savings because of higher density hardware, in which one single appliance has double throughput compared to previous generation, and more. To know more about AEWIN PCIe Gen 5 based extraordinary products and customizations, please don’t hesitate to contact AEWIN friendly sales!

– SCB-1836: Alder Lake-S/Raptor Lake-S PCIe 5.0 Platform with 2x PCIe Gen5 slots plus 2x PCIe Gen4 NIC, IPMI, and TSB module.

– SCB-1941: Sapphire Rapids-SP PCIe 5.0/CXL Platform with short depth design, 4x PCIe Gen5 slots NIC, NVMe, and IPMI.

– SCB-1942: Dual Sapphire Rapids-SP PCIe 5.0/CXL Platform with short depth design, 4x PCIe Gen5 slots plus dual FHFL GPU slots or 4x PCIe Gen4 NIC, and IPMI.

– SCB-1943: Sapphire Rapids-SP PCIe 5.0/CXL Platform with short depth design, 4x PCIe Gen5 slots plus dual FHFL GPU slots or 4x PCIe Gen5 NIC, and IPMI.

– SCB-1946: Dual EPYC-9004 (Genoa/Bergamo) PCIe 5.0/CXL Platform with short depth design, 4x PCIe Gen5 slots plus dual FHFL GPU slots or 4x PCIe Gen4 NIC, and IPMI.

– BAS-6101: EPYC-9004 (Genoa/Bergamo) PCIe 5.0/CXL Platform with short depth design, 6x Full Height PCIe slots plus 2x Half Height slots, 1x OCP3.0 slot, and optional 2x Dual Width slots.