Abstract: The rapid growth of Internet traffic has placed higher demands on backbone network transmission. To address this, we developed a high-performance IPv4/v6 router with a total throughput of up to 1.28Tbit. This router provides interfaces for 2.5GPOS, 10GPOS, 100M Ethernet, 1000M Ethernet, 10GLAN Ethernet, and 10GWAN Ethernet, facilitating easy network deployment. This paper briefly introduces the overall structure of this router and delves into the design and implementation of the 2.5GPOS interface board.
Keywords: router; synchronous optical network; synchronous digital series; embedded system
1. Introduction
As the hub of the Internet, routers are evolving in three directions: faster speeds, better service quality, and easier integrated management. Currently, chip speeds double every 18 months, while Internet traffic doubles every six months. Research on routers has focused on improving their processing speed, leading to the development of Gigabit Switch Routers (GSRs) and Terabit Switch Routers (TSRs).
As the core position of IP is gradually recognized, the IP over ATM, and then ATM over SDH approach has been replaced by IP directly over SDH. Transmission speeds have also rapidly jumped from OC-12 (622Mbps) to OC-48 (2.5Gbps). POS technology is now widely used in backbone networks, and the POS interface has become an essential component of high-end routers.
The T-bit router I developed achieves a total system throughput of 1.28Tbps, a peak switching capacity of 2.4Tbps, and a forwarding rate of 10G per port. To achieve the 10G forwarding rate, this router employs four 2.5GPOS interfaces. Section 2 describes the hardware architecture design and performance requirement analysis of the 2.5GPOS interfaces, while Section 3 details the software architecture design and implementation.
2. Overall Hardware Structure Design
The hardware architecture of this router consists of four main parts: the line interface subsystem, the forwarding processing subsystem, the optical backplane switching network subsystem, and the internal communication subsystem. When an interface receives a data packet, it first determines whether the destination address of the packet belongs to this router. If so, it hands the packet over to the single-board software and then to the main control unit via the internal communication subsystem; otherwise, it fills in an internal header, indicating the packet type, protocol type, destination rack number, port number, interface number, etc., and hands it over to the forwarding processing subsystem via coaxial cable, and then forwards it to the destination interface via the optical backplane. The interface then performs routing processing on the incoming packet and the packet sent by this router before sending it out again.
2.1 Performance Requirements Analysis
1) Based on past experience with routers, high-speed channels in hardware FPGAs typically operate stably at 2.5Gbps. For a 10Gbps input data rate on port A, four 2.5Gbps high-speed channels are sufficient to transmit data to the forwarding processing module. However, the line interface module requires that data packets have a 128-bit (16-byte) internal tag added after processing. Considering the extreme case where port A receives ultra-short 40-byte packets, the maximum output data rate of port B is: [10Gbps × (40 bytes + 16 bytes)] / 40 bytes = 14Gbps. Therefore, we design to use two 2.5Gbps high-speed channels to transmit one-quarter (2.5Gbps) of the 10Gbps data received by port A to the forwarding processing module. Thus, the maximum output rate of port B is 2.5Gbps × 8 = 20Gbps, fully meeting the bandwidth requirements of port B.
2) Under the uniform input and balanced output service models, assuming a fixed-length packet service intensity λ of 0.95, and to control the packet loss probability to less than 10⁻⁸, the required output queue packet buffer capacity S is approximately 650 packets. Statistically, the average length of an IP packet is 508 bytes. Considering the 24 bytes added by the internal packet label, the average number of IP packets that need to be buffered is 532 bytes. The total buffer capacity is 650 × 532 × 8 = 2,766,400 (Bits), which is approximately 2.8 MBits.
3) The interface speed of this subsystem needs to reach 10Gbps. If each packet is 40 bytes, then the maximum table lookup rate is:
Another key metric for the input lookup module is the table entry capacity. Based on the maximum capacity, a T-bit router has 8 racks, each rack has 8 ports, each port has 8 interfaces, and each interface has a total of 64 IPv4 unicast/multicast and IPv6 unicast/multicast addresses. Therefore, the maximum table entry capacity is 8 * 8 * 8 * 64 = 32768 (32k) entries. Achieving these two design metrics was a challenge in this module's design. Ultimately, we selected a CAM with 64K entries and a lookup rate of 100MSPS. This metric fully meets the requirements of this module.
2.2 Subsystem Function Division and Module Design
As shown in the figure below, the 4 2.5G line interface subsystem can be divided into an optoelectronic and serial-to-parallel conversion module, a link layer processing module, an output FPGA processing module, an input FPGA processing module, a protocol message submission buffer module, an input lookup table module, and a clock processing module. For ease of description, the interface between this subsystem and the router is referred to as port A, the interface with the forwarding processing subsystem as port B, the interface with the optical backplane subsystem as port G, the interface with the single-board processor as port F, and the interface with the clock subsystem as port J.
When receiving data from outside the router, the optoelectronic and serial-to-parallel conversion module receives four 2.5Gbit/s serial optical signals from port A and converts them into 16 differential 622MHz electrical signals; these are then sent to the link layer processing module for link layer processing. The link layer processing module extracts PPP packets from the SDH frames and passes them to the input FPGA processing module. The input FPGA processing module looks up the local IP address table stored in the input lookup module based on the type of the input packet, performs demultiplexing, and if the packet's destination is this router, it is forwarded to the main control; otherwise, it is forwarded to port B. The output FPGA processing module receives protocol packets from the main control and forwards G-port data packets from the system optical backplane, combines them, and then passes them to the link layer processing module. The link layer processing module encapsulates the combined PPP packets in an SDH frame, which is then converted into optical signals by the optoelectronic and serial-to-parallel conversion module and sent out. The clock processing module provides a global clock for this module and outputs the clock extracted from the line to the clock board.
3. Software Overall Structure Design and Implementation
The MPC860 is a communication controller based on the PowerPC architecture. It is not only an integrated microprocessor but also integrates the functions of many peripherals. The MPC860 has a powerful memory controller that supports various types of memory, including newer DRAM and Flash memory, and can achieve seamless interfacing with the memory.
VxWorks, an embedded real-time operating system developed by Wind River Systems, gained fame for its use on Mars rovers. VxWorks is a microkernel-based, high-performance, scalable operating system that supports a wide range of network communication protocols and can be customized to meet user needs. Combined with its excellent Tornado development environment, it has become a top choice for embedded operating systems. Choosing the MPC860 with the VxWorks embedded real-time operating system is perfectly adequate for the performance requirements of a single-board processor.
The diagram above illustrates the overall architecture of the single-board software. The single-board operating system is VxWorks, and the single-board processor is MPC860. Routers have different types of line interface boards, each requiring many of the same functions. These common functions can be implemented in a separate module, reducing redundant work. The board-level support module provides services such as communication with the main control software, task management, memory management, and timer management for the single-board application; while the single-board application, the line interface single-board software module, handles the single-board forwarding processing.
The main functions of the single-board software include: initializing and configuring various types of single-board hardware, receiving and processing single-board control commands issued by the main control unit, loading software and hardware programs, running the IPv4 ARP protocol on the Ethernet interface board, maintaining and announcing ARP information to the main control unit, sending and receiving protocol data (packets with source/destination addresses of this router), maintaining router interface address information, handling error messages, recording message reporting, collecting and reporting service performance statistics data, and maintaining the communication connection between the single board and the main control board.
Based on these functions, the single-board software is divided into an initialization module, a main control command processing module, an interface chip monitoring and service performance statistics module, a message distribution module, a packet sending module, a packet receiving module, a self-test module, a CAM table maintenance module, and a utility function module. During normal operation, four tasks are established: a main control processing task, a statistics task, a packet sending task, and a packet receiving task.
The data processing flow is shown in the diagram on the right: The single-board hardware receives the packet from this router, writes it into the input FIFO, the packet receiving task reads the packet, encapsulates it in the defined internal data structure, and calls the sending function in the board-level support module to send it to the main control; the packets sent by the main control are processed by the message distribution module, the protocol packets are handed over to the packet sending task, then written into the output FIFO, and then sent out, and the control packets are handed over to the main control processing module for processing to complete the corresponding functions.
Because this interface requires a speed of 10Gbps, a stable and high-speed interaction mechanism is essential between tasks. The VxWorks operating system provides various communication methods for inter-task communication, including shared memory, semaphores, message queues and pipes, sockets, and remote calls. Message queues allow for variable-length and variable-number message queues, satisfying communication needs between multiple tasks. The VxWorks Wind kernel can employ a combination of priority-based preemptive task scheduling and round-robin scheduling. Therefore, it is crucial to properly configure the priorities of each task; for example, memory tasks must have a higher priority than the master processing task, and the master processing task must have a higher priority than packet sending and receiving tasks. For tasks with the same priority, their execution time must be pre-set; otherwise, the first task to acquire the CPU will monopolize it, hindering the operation of other tasks. In short, a thorough understanding of the VxWorks working mechanism is essential in software design; otherwise, not only will efficient programs not be designed, but the effective operation of VxWorks may also be impeded.
4. Conclusion
The rapid growth of network and internet traffic has placed higher demands on existing network infrastructure and stimulated the development of high-performance network connections. This paper briefly introduces a high-performance router with a throughput of up to 1.28 Tbits and studies the implementation technology of its 2.5G POS interface. Firstly, this allows for in-depth research into the implementation and application of POS technology; secondly, it enriches the router's interface functionality, increases its flexibility, and enhances its compatibility, enabling wider utilization of network resources and facilitating the evolution of network architecture. This provides a valuable technical approach for the research and development of high-performance routers and their subsequent advancements.
References:
[1] Yang Degui et al., Network and Broadband IP Technology, China Electric Power Press, 2002, 11.
[2] Xue Hongxi, Bian Jinian, Su Ming, Digital System Design and Automation, Tsinghua University Press, 1996, 10.
[3] Zhou Qiping, Zhang Yang, Wu Qiong, VxWorks Development Guide and Tornado Practical Manual, China Electric Power Press, 2004, 10
[4] Zhou Qiping, Device Driver and BSP Development Guide under VxWorks, China Electric Power Press, 2004, 8
[5] Kong Xiangying, Bai Guizhi, Embedded Real-Time Operating System VxWorks and its Development Environment Tornado, China Electric Power Press, 2002, 1
