This article is from "The First Year of AI Servers: An Overview of Interface Interconnect Chips". According to TrendForce data, approximately 130,000 AI servers were shipped, accounting for about 1% of global server shipments. Looking ahead to 2023, Microsoft, Meta, Baidu, and ByteDance have successively launched products and services based on generative AI and are actively increasing orders. It is estimated that the annual growth rate of AI server shipments in 2023 will reach 15.4%. Furthermore, with the future demand for AI servers from ChatGPT, AI servers are expected to grow at a CAGR of 12.2% from 2023 to 2027.
Launched in 2022, the DGX H100 is the latest iteration of NVIDIA's DGX system and the foundation of the NVIDIA DGX SuperPOD. The DXG server is equipped with eight H100 GPUs, 640 billion transistors, and delivers 6 times the AI performance of its predecessor at the new FP8 precision, providing 900GB/s of bandwidth.
Inside the DGX H100 server, the blue square represents the IP network interface card (NIC), which functions as both a network card and a PCIe switch, acting as a bridge between the CPU and GPU (H100). It internally adopts the PCIe 5.0 standard. Furthermore, the CX7 chips are integrated into two boards inserted into the server, with four CX7 chips forming one board and providing two 800G OSFP optical module ports.
The interconnection between GPUs (H100) is primarily achieved through NV Switch chips. Each GPU within the DGXH100 extends 18 NVLinks, with a single-link bidirectional bandwidth of 50 GB/s, totaling 18 * 50 GB/s = 900 GB/s bidirectional bandwidth. This bandwidth is distributed across four onboard NV Switches, with each NV Switch corresponding to 4-5 OSFP optical modules (a total of 18). Each OSFP optical module uses 8 optical channels, with each channel having a transmission rate of 100 Gbps, resulting in a total rate of up to 800 Gbps, enabling high-speed data transmission.
1. Interconnection of components such as CPU and GPU: PCIe switch, Retimer chip
A PCIe switch, also known as a PCIe switch or PCIe adapter, primarily enables the interconnection of PCIe devices. The communication protocol between the PCIe switch chip and the devices is PCIe. Because PCIe link communication is an end-to-end data transmission, the switch needs to provide expansion or aggregation capabilities to allow more devices to connect to a single PCIe port, thus addressing the issue of insufficient PCIe lanes. Currently, PCIe switches are not only widely used in traditional storage systems but are also gradually becoming more common in some server platforms to improve data transmission speeds.
PCIe bus technology upgrades, with each generation of PCIe switches increasing speed. The PCIe bus is a high-speed serial replacement for the PCI bus. In 2001, Intel announced its third-generation I/O technology, "3GIO," to replace the PCI bus. In 2002, after review by the PCI Special Interest Group (PCI-SIG), this technology was officially renamed "PCI Express," marking the birth of PCIe. In 2003, PCIe 1.0 was officially released, supporting a transfer rate of 250MB/s per lane and a total transfer rate of 2.5 GT/s. In 2007, the PCI-SIG announced the PCI Express Base 2.0 specification. Based on PCIe 1.0, it doubled the total transfer rate to 5 GT/s, and increased the transfer rate per lane from 250 MB/s to 500 MB/s. In 2022, the PCI-SIG officially released the PCIe 6.0 specification, increasing the total bandwidth to 64 GT/s.
With the increasing adoption of PCIe in servers, the market demand for PCIe switches has also surged. According to statistics and forecasts from QYResearch, the global PCIe chip market reached $790 million in sales in 2021 and is projected to reach $1.8 billion by 2028, representing a compound annual growth rate (CAGR) of 11.9%.
China is the largest market for PCIe switches. With servers increasingly demanding massive data storage and transmission capabilities, the PCIe switch market has become a blue ocean. China is one of the world's largest producers and consumers of electronic products, and its big data, cloud computing, and artificial intelligence sectors require numerous high-speed interconnect solutions to achieve massive data transmission. As a high-speed interconnect solution, PCIe switches have enormous demand in the Chinese market.
In AI servers, at least one retimer chip is required to ensure signal quality when the GPU and CPU are connected. Specifically, many AI servers are configured with multiple retimer chips; for example, Astera Labs uses four retimer chips in its AI accelerator.
The PCIe Retimer market is a three-way competition, with potential rivals eager to enter. Currently, Parade Technologies, Astera Labs, and Montage Technology are the three major manufacturers in this market, holding leading positions. Montage Technology, in particular, entered the PCIe market early and is the only mainland Chinese supplier globally capable of mass-producing PCIe 4.0 Retimers. Furthermore, its PCIe 5.0 Retimer development is progressing smoothly.
In addition, chip manufacturers including Renesas, TI, and Microchip Technology are also actively investing in the research and development of PCIe Retimer products. According to their official websites, Renesas offers two PCIe 3.0 Retimer products, namely the 89HT0816AP and 89HT0832P; TI offers a 16Gbps 8-lane PCIe 4.0 Retimer—the DS160PT801; similarly, Microchip Technology released its XpressConnect series of Retimer chips in November 2020, which supports PCIe 5.0 speeds of up to 32GT/s.
2. GPU-to-GPU connections: NVLink, NVSwitch
Major global chip manufacturers are paying attention to high-speed interface technologies. In addition to NVIDIA's NVLink, AMD's Infinity Fabric and Intel's CXL (Compute Express Link) also provide solutions for high-speed interconnection within servers.
The continuously iterating and updating NVLink is sparking a revolution in high-speed interconnect technology. NVLink is a high-speed interconnect technology developed by NVIDIA, designed to accelerate data transfer between CPUs and GPUs, and between GPUs, thereby improving system performance. From 2016 to 2022, NVLink has been iterated to its fourth generation.
In 2016, NVIDIA released NVLink, a new high-speed interface chip for the Pascal GP100 GPU. This was the first generation of NVLink. NVLink uses High-Speed Signaling Interconnect (NVHS) technology and is mainly used for signal transmission between GPUs and between GPUs and the CPU. It transmits differential impedance electrical signals in NRZ (Non-Return-to-Zero) encoding between GPUs. The first-generation NVLink could achieve a bidirectional bandwidth of 40 GB/s per link, and a single chip could support four links, for a total bidirectional bandwidth of 160 GB/s.
Since then, NVLink has undergone multiple iterations and updates, sparking a wave of innovation in high-speed interconnect technology. In 2017, the second-generation NVLink, based on the Volta architecture, was released, achieving a bidirectional bandwidth of 50 GB/s per chain, with a single chip supporting 6 chains, or a total bidirectional bandwidth of 300 GB/s. In 2020, the third-generation NVLink, based on the Ampere architecture, was released, achieving a bidirectional bandwidth of 50 GB/s per chain, with a single chip supporting 12 chains, or a total bidirectional bandwidth of 600 GB/s. In 2022, the fourth-generation NVLink, based on the Hopper architecture, was released, changing the transmission signal to PAM4 modulated electrical signals, achieving a bidirectional bandwidth of 50 GB/s per chain, with a single chip supporting 18 chains, or a total bidirectional bandwidth of 900 GB/s.
In 2018, NVIDIA released its first-generation NVSwitch, providing a solution for improving internal server bandwidth, reducing latency, and enabling communication between multiple GPUs. The first-generation NVSwitch was manufactured using TSMC's 12nm FinFET process and featured 18 NVLink 2.0 interfaces. A single server could support 16 V100 GPUs through 12 NVSwitches, achieving interconnection at the highest speeds of NVLink.
The NVSwitch has now reached its third generation. The third-generation NVSwitch is built using TSMC's 4N process, with 64 NVLink 4.0 ports on each NVSwitch chip. The inter-GPU communication rate can reach 900GB/s. These GPUs interconnected through the NVLink Switch can be used as a single high-performance accelerator with deep learning capabilities.
3. High-speed interconnection between the CPU and DRAM drives the memory interface chip.
Server memory modules are primarily of two types: RDIMM and LRDIMM. Compared to other types of memory modules, server memory modules have higher requirements for stability, error correction capabilities, and low power consumption. The memory interface chip is the core logic device of the server memory module and is the essential path for the server CPU to access memory data. Its main function is to improve the speed and stability of memory data access, meeting the ever-increasing high-performance and high-capacity demands of server CPUs for memory modules.
From DDR4 to DDR5, memory interface chip speeds have continuously upgraded. Starting in 2016, DDR4 became the mainstream technology in the memory market. To achieve higher transfer rates and support larger memory capacities, the JEDEC organization further updated and improved the technical specifications of DDR4 memory interface chips. In the DDR4 generation, from Gen1.0, Gen1.5, Gen2.0 to Gen2plus, the maximum transfer rate supported by each generation of memory interface chips has continuously increased. The last generation of DDR4, Gen2plus, supports a maximum transfer rate of 3200MT/s. As the JEDEC organization continues to refine the specifications for DDR5 memory interface products, DDR5 memory technology is gradually replacing and updating DDR4 memory technology.
Currently, three generations of DDR5 memory interface chips have been planned, supporting speeds of 4800MT/s, 5600MT/s, and 6400MT/s respectively. Industry insiders expect that there may be one or two more generations in the future.
Memory interface chips are functionally divided into two categories: Register Buffers (RCDs) and Data Buffers (DBs). RCDs are used to buffer address, command, and control signals from the memory controller, while DBs are used to buffer data signals from the memory controller or memory chips.
The upgrade to DDR5 memory modules brings new opportunities for memory interface chips and related chips. The global memory interface chip market was worth approximately $280 million in 2016 and reached approximately $570 million in 2018, representing a three-year annualized growth rate of 40%. The DDR5 upgrade will drive the expansion of the memory interface chip market. Compared to DDR4, due to DDR5's higher supported speeds and more complex design, the starting unit price of first-generation DDR5 memory interface chips is higher than that of DDR4 memory interface chips. Simultaneously, as the penetration rate of DDR5 memory in servers and PCs gradually increases, the market size of DDR5-related memory interface chips is expected to achieve rapid growth.
The memory interface chip industry has high barriers to entry, and a three-way competition has already formed. Memory interface chips are a technology-intensive industry, requiring rigorous verification from CPU, memory, and OEM manufacturers before large-scale use, making it difficult for new players to enter. As the technological difficulty continues to increase, the number of players in the memory interface chip market has dwindled from over 10 in the DDR2 generation to only 3 in the DDR4 generation, essentially eliminating competitors and establishing a three-way competition. In the DDR5 generation, only three global suppliers can provide mass-produced first-generation DDR5 products: Montage Technology, Renesas Electronics (IDT), and Rambus.
