summary
Modern electronic systems require increasingly larger volumes of data, including both analog and digital data. Therefore, the biggest challenge for researchers and system developers is integrating, testing, and maintaining faster and more accurate test systems to handle the ever-increasing data volumes. For certain applications, such as video signal analysis, the requirement is to continuously acquire high-speed analog and digital information for real-time analysis. Therefore, when designing or selecting such measurement systems, a clear understanding of data flow and potential problems can reduce development costs, accelerate time-to-market, and avoid costly system redesigns.
Applications such as spectrum monitoring, signal analysis, LiDAR signal acquisition, fiber optic testing, and radar and satellite signal collection are typical examples of high-speed and high-precision data acquisition applications. The biggest challenge for engineers in these applications is meeting the high-bandwidth system requirements. This article will share details to consider when building high-bandwidth systems using the PXI Express platform, such as the onboard memory of digitizers or waveform generators, the PCIe signal architecture within the PXI Express chassis, the computer operating system, and the selection of memory and storage devices.
Introduction
Traditional desktop instruments use GPIB, RS-232, or LAN interfaces for data transmission. While these interfaces are easy to use, their efficiency is less than ideal when transmitting large amounts of data. When acquiring large amounts of continuous data, the data length is limited by the size of the internal memory. Newer, high-end instruments on the market, such as oscilloscopes, waveform generators, and logic analyzers, use the x86 architecture, thus having virtually no limitations on the length of high-speed, large-volume data acquisition. However, achieving simultaneous multi-channel acquisition across multiple instruments presents a difficult and complex challenge.
Since the first version of the PXI specification was released in 1998, the PXI platform and its modules have been widely used in military, electronic manufacturing, and scientific research applications. The first version of the PXI specification adopted the high-speed transmission characteristics of the PCI bus, while subsequent PXI specifications adopted the PCI Express bus, inheriting its low latency, high bandwidth, and point-to-point transmission characteristics. In addition, the unique trigger and timing synchronization interface makes the PXI platform and PXI modules particularly suitable for high-speed data transmission.
When designing a high-speed data logging system using the PXI platform, whether continuously transferring data from modular instruments to system memory or storage devices, or vice versa, the high-speed bus, point-to-point transmission capabilities, and unique triggering and timing signals of PXI Express can be easily achieved. The following sections will further discuss several key considerations and directions for designing and implementing data logging systems.
The architecture of a data recording system and its considerations
Figure 1 below illustrates the data flow within a PXI Express platform. The components include a PXI Express chassis, a PXI Express controller, and modular instruments, including digitizers and waveform generators. Taking a high-speed digitizer as an example, after the analog signal is acquired by the ADC and converted into digital data, it is temporarily stored on the onboard memory. Then, via the bus controller and PCI Express interface, it is transmitted to the system memory of the PXI Express controller for further calculations and processing. If the data's destination is a storage device, it is directly moved to the storage device without any processing or calculation, maintaining high-speed, continuous data recording. The PXI Express backplane uses PCIe switches to allow for the expansion of more slots. Because different PXI Express chassis have different slot configurations, the wiring method of each PCIe switch is different, thus affecting data transmission efficiency. For example, the data flow of a modular instrument—the waveform generator—runs in the opposite direction.
Figure 1. Simplified diagram of the overall architecture of the PXIExpress platform and modular instruments, showing the direction of data recording and transmission in the PXIExpress system.
Next, we will discuss each aspect of the data logging system and its impact on data bandwidth.
Onboard memory of modular instruments
About a decade ago, high-speed PCI digitizers required substantial onboard memory to temporarily store data from high-speed ADCs. This was primarily because the PCI bus at the time only offered a data bandwidth of approximately 132 MB/s (most systems only reached about 80 MB/s). The PCI bus bandwidth was insufficient for the data bandwidth required by 8-bit 1 GS/s or 14-bit 200 MS/s digitizers. To increase recording or acquisition time, 512 MB, 1 GB, or even 4 GB of onboard memory were used in digitizers. Currently, although the high-speed PCIe bus interface offers several times the bandwidth of the PCI bus, digitizers still typically have a significant amount of onboard memory (greater than 100 MB) for temporary data storage to prevent the CPU or DMA controller from becoming too busy to transmit data in real time. For example, a single-channel 8-bit 500 MS/s digitizer could record for up to 1 second without transferring data back to system memory, and up to 4 seconds with 2 GB of memory.
When selecting a digitizer, another important factor to consider is the data processing bandwidth of its onboard memory controller. As a bridge between the ADC and system memory, the memory controller needs twice the data transfer capacity to simultaneously handle the data inflow from the ADC and the data transfer to system memory via the PCIe bus. If the memory controller's bandwidth is less than twice the data flow, data will be temporarily stored on the onboard memory, eventually leading to data overflow and impaired data continuity.
Figure 2. Data flow within the digitizer
Bus interface of modular instruments
The PCI bus offers a transfer rate of 132MB/s (32-bit, 33MHz), which is sufficient for low-speed (less than 80MB/s) and low-cost data recording applications. However, it's important to note that PCI is a parallel bus interface; if multiple devices are placed on the same bus, the bandwidth will be shared. Unlike the PCI bus, the PCI Express interface is point-to-point, with each link providing up to 250MB/s transfer rates in each direction. To increase bandwidth, the simplest way is to combine multiple links to create x4, x8, or even x16 channels. The PCIe 1.0a specification was introduced in 2003, followed by the PCIe 2.0 standard in 2007. In November 2010, the PCI-SIG introduced the PCIe 3.0 specification, continuously pushing forward with updated encoding methods and enhanced signal integrity to significantly improve transfer rates. Therefore, modular instruments using the PCIe interface are a major advantage for applications requiring high-speed data recording. Clearly, using PCI Express as the bus interface for modular instruments can achieve optimized system efficiency.
The PCIe bus routing architecture within the PXI Express chassis
In PXI Express chassis, to allow for greater flexibility in the expansion and planning of peripheral slots, the interfaces connecting the system slots to the backplane offer both 4-Link and 2-Link architectures. In the 4-Link architecture, each link has 4 channels, while the 2-Link architecture allows one link to have 8 channels, and another link to have up to 16 channels. To achieve the highest transmission rates, the routing and architecture of the PCI Express bus within the PXI Express chassis are also crucial considerations. Taking the ADLINK PXES-2780 chassis as an example, this is an 18-slot chassis, including 1 system slot, 1 system timing slot, 6 PXIe peripheral slots, and 10 hybrid slots. When the system slots of this chassis are configured with 4-Link interfaces, it can provide relatively high and balanced transmission rates for each slot. Because the PCI Express interfaces in this chassis are PCIe Gen2, they can provide up to 8GB/s of system bandwidth for the entire system. Slots 8 and 12, which have x8 interfaces, can provide 4GB/s of bandwidth, while the other individual PXI Express peripheral slots can provide 2GB/s of bandwidth. The 4-Link layout diagram of this chassis is shown below:
Figure 3. Schematic diagram of ADLINK PXES-2780 chassis configured with 4-Link.
Setting the PXIExpress system slots to 2-Link x8 provides higher bandwidth. ADLINK's PXES-2780 chassis allows system slots to be configured with 2-Link x8 interfaces via software, as shown in Figure 4. Using this architecture, slots 8 and 12 can provide x8 of the bandwidth.
Figure 4. Schematic diagram of ADLINK PXES-2780 chassis planned for 2-Link configuration.
Users familiar with the PXI Express chassis architecture will be able to achieve better transmission performance for modular instruments when transmitting large amounts of data.
System memory and operating system (OS)
On a PXI Express system controller, a large amount of system memory can extend data logging time. However, different operating systems may have different memory limits. For example, a 32-bit operating system typically has a memory addressing space of no more than 4GB, while a 64-bit operating system can generally support up to 512GB or 1TB of memory addressing. Therefore, users need to choose a suitable operating system based on their specific needs to support the required memory space.
storage device
Choosing the right storage device is crucial for stable large-scale data read and write operations. A hard disk drive (HDD) is a specialized mechanical device containing a high-speed rotating disk and a magnetic read/write head that moves back and forth across the disk surface to read the data stored on it. Therefore, read or write speeds are limited by the speed of the read/write head's movement. To increase read and write speeds, several HDDs are often combined into a virtual hard drive, a feature known as RAID (Redundant Array of Independent Disks). Solid-state drives (SSDs), which are becoming increasingly popular, offer even better read and write efficiency than HDDs because they eliminate the mechanical movement of the read/write head. Considering optimal read and write performance, SSDs are the best choice.
Application Example 1: Recording Data to System Memory
High-speed data recording for material structure testing
Solution Requirements
A system integrator wants to develop a multi-channel material vibration monitoring device capable of simultaneously acquiring data from different sensors at sampling rates from 1MS/s to 50MS/s, and recording the acquired data into system memory for direct processing without storing it on disk. Furthermore, the integrator requires each acquisition channel to record for at least 5-10 seconds. We will evaluate these application requirements and discuss the design bottlenecks encountered at different sampling rates when implementing this type of application using the PXI Express platform.
Evaluate
Below are the components we will use to evaluate this high-speed data logging system:
PXI Express Chassis: ADLINK PXES-2780, 18-slot PXI Express Chassis
Digitizer: ADLINK PXIe-9848, high-speed 8-channel 100MS/s 14-bit PXI Express digitizer
First, let's consider the scenario where the PXI Express chassis contains only a single digitizer. The table below shows the memory requirements of a single digitizer at different sampling rates and acquisition times.
When using only one PXIe-9848 digitizer, sampling at 100MS/s for 8 channels results in a total data bandwidth of 1.6GB/s. However, the PXIe-9848 uses a PCIe x4 interface (Gen1), so for continuous data recording, the data volume should ideally be below 1GB/s. Reducing the sampling rate to 50MS/s allows the PXIe-9848 to generate 800MB/s of data. Since the PXES-2780 chassis uses a PCIe Gen2 form factor, it can handle 800MB/s of data. For 10-second sampling at 50MS/s, a further limitation is placed on system memory size, requiring 8GB of system memory. If the system memory cannot allocate up to 8GB for the digitizer, the sampling time must be shortened.
Next, let's consider the scenario where multiple cards are installed in the same system. To achieve the maximum number of sampling channels, up to 17 PXIe-9848 digitizers can be installed in the chassis. In this case, in addition to considering the system bandwidth of the PXIExpress controller, the PCIe connection configuration on the PXIExpress backplane must also be taken into account.
Observing the connection configuration of the PXES-2780 chassis, as shown in Figure 4, its PCIeswitch #1 has two x4 connection channels upstream and three x4 connection channels downstream. Therefore, on average, each downstream channel can obtain approximately 1.33GB/s of bandwidth (4GB/s ÷ 3 ports) from the upstream channels. The downstream bandwidth of PCIeswitch #2 can be calculated in the same way. As for PCIeswitch #3 and #4, their downstream channels can obtain approximately 222MB/s (1.33GB ÷ 6 ports) and 190MB/s (1.33GB/s ÷ 7 ports) of bandwidth, respectively. If each digitizer uses the same sampling rate, the bandwidth bottleneck will appear on the digitizers located in slots 10 and 13-18.
Figure 5. Estimated bandwidth diagram of a chassis configured as PCIe x4link
If we calculate the system memory required for different sampling time lengths based on the bandwidth limitations of slots 10 and 13-18, the results are shown in the table below:
The evaluation results in the table above show that if continuous sampling is performed for 5 seconds at a sampling rate of 5 MS/s, the 17 digitizers will require a total system memory of 6.8 GB. If the sampling rate reaches 10 MS/s, the recording time will be reduced to 2 seconds (requiring 5.44 GB of memory). If a longer recording time is still needed, the onboard memory of the digitizer can extend the recording time to a certain extent.
Application Example 2: Recording Data to Disk
Signal acquisition of high-speed photodiodes in laser monitoring
Solution Requirements
The customer's application involves laser signal monitoring, requiring the acquisition of photodiode signals in a portable design. Only one channel needs to be acquired, but the sampling rate must be as high as 200 MS/s.
Solution
Because the customer required a single-channel sampling rate of up to 200MS/s, the application combination used was the ADLINK PXIe-9842 and the portable PXIExpress chassis PXES-2590, providing a data recording rate of up to 400MB/s. Since data needed to be recorded to disk, and the disk devices on the PXIExpress controller generally cannot provide bandwidth up to 400MB/s, even with SSDs, an external RAID storage device was used for data access. The RAID module we used was a PXIExpress interface SSD with four SATAIII interfaces.
PXI Express System Controller: ADLINK PXIe-3975, 3U Intel® Core™ i5-520E 2.4GHz Dual-Core PXI Express System Controller
PXI Express Chassis: ADLINK PXES-2590, 9-slot hybrid PXI Express chassis
Digitizer: ADLINK PXIe-9842, a 14-bit PXI Express digitizer with a sampling rate of 200 MS/s.
RAID storage device: ConduantDM-425
in conclusion
To implement data logging applications using the PXI Express platform, it's necessary to consider not only the capabilities of the modular instrument itself but also the data transmission bandwidth of the PXI Express platform. Through the design details and application examples discussed in this article, users will be able to effectively build higher-performance measurement and testing platforms and significantly improve development efficiency.
About Linghua
ADLINK Technology provides Application-Ready Intelligent Platforms (ADIPs) for measurement and testing, industrial automation, network communications, military, transportation, medical, and infotainment industries, offering innovative embedded computing solutions. ADLINK is a Premier Member of the Intel® Internet of Things Solutions Alliance, a member of the PICMG® Association and the PC/104 Association (capable of participating in specification development), a Board Member and Top-Level Member of the PXI Systems Alliance (PXISA), a Strategic Member of the AXIe Alliance, a member of the VMEbus International Trade Association (VITA), and a member of the SGET (Simplified Form Engineering Technology) organization. Headquartered in Taiwan, ADLINK has manufacturing centers in Taiwan and mainland China, while its R&D and integration business groups are located in Taiwan, China, the United States, and Germany, with sales and service locations worldwide. ADLINK is ISO 9001, ISO 14001, ISO 13485, and TL9000 certified, providing reliable products, fast service, and real-time support to customers worldwide. Website: http://www.adlinktech.com/cn
