Modern microelectronics has opened up new possibilities for the development of distributed systems in experimental physics, engineering, and other applications. New modular microprocessor systems for data acquisition (DAQ) and network interconnection, as well as distributed control applications, are being proposed and discussed. High-power, multi-functional specialized microprocessors (for communication and DSP) are operating as real-time systems with distributed memory. Embedded modular systems for digital signal processing (DSP) and control applications are being developed in industry standards (Industrial Computer System (ICS) ISA chassis (MicroPC) with two/four slots or cPCI with four to eight slots). Multiprocessor systems are effectively applied to distributed control, and real-time multiprocessor cores and operating systems (OS) for these systems are being researched to adapt to various applications in experimental research and engineering. The idea of ​​connecting compact industrial computer systems (ICS) with active backplanes and compact PCI-based systems (cPCI/PXI) with passive backplanes via Ethernet as embedded real-time systems for control applications is being considered. People are also considering connecting distributed systems with System Area Networks (SANs) as advanced modular systems with parallel pipelined data processing capabilities for data acquisition and control applications. Based on joint modeling of the data acquisition, triggering, and control subsystems, the integration of experimental physics and engineering subsystems is being considered. 1. Embedded Modular Real-Time Systems with Active Backplanes Microcomputers consist of many components (inserted into a backplane), including a CPU, memory, disk drives, and serial/parallel ports. Some computers are based on IBM PC (ISA bus) plug-in modules, others are implemented as stand-alone systems on a single board (backplane-less), and still others are single-board computers (SBCs) based on backplane buses (VME/VXI). Backplane-based microcomputers can be used for data acquisition, process control, and various R&D projects, but generally, due to their large size, they are not embedded as intelligent components in devices. In the 1980s, computer boards were limited by large-scale integrated chips; integrated circuits, due to their advanced performance, dominated the entire computer board market, later evolving into microcontrollers or DSPs. The PC/104 and PC/104-Plus modules tend to consist of standard PC desktop and laptop components that support embedded Linux. The PC/104-Plus adds a PCI bus using a board-to-board (120-pin) bus. There is growing interest in IBM PC compatibility for PC-based non-desktop embedded systems: • PC chip-level and peripheral compatibility enables lower costs, simpler architectures, and easier support; • PC compatibility offers advantages in terms of PC operating systems (MS-DOS, Windows, Linux), languages, and tools. The emergence of new interfaces (USB, FireWire, Bluetooth), architectures (MIPS, PowerPC, ARM), and operating systems (RTLinux, RTEMS) has enabled embedded single-board computer (SBC) platforms to better serve embedded modular real-time systems: • Increased embedded intelligence, with many applications requiring user-friendly graphical and voice interfaces; • Increased demand for interconnected electronic devices (TCP/IP, PPP, HTTP, FTP); • USB is replacing serial, parallel, and PS/2 interfaces, Ethernet is ubiquitous, and FireWire (IEEE-1394) is beginning to be used; • Processors (highly integrated application-oriented system-on-a-chip based on ARM, MIPS, PowerPC, and x86) are under development; Linux is used for all computational processing, providing a low-cost, open-source solution that supports open standards, networking, communication, the Internet, and other functions. A proposal has been made to connect compact, modular systems with two slots based on Small Industrial Computer Systems (ICS) as embedded controllers (CS) and virtual workstations (VS) in a distributed network via 10/100M Ethernet. Each virtual terminal (VS) is based on Windows and/or Linux, and each controller terminal (CS) is based on RT-Linux and used for data acquisition, monitoring, and control. One of the two PCI slots is used for DSP-based data acquisition and control modules, and the other is used for expansion or additional Ethernet connections. Typically, fieldbus utilizes an economical modular approach in both hardware and software to achieve different application outcomes. Today, most computers use traditional networks (10/100M Ethernet, FireWire, USB) as the standard connection. The concept of fieldbus should be transparent to all electronic devices. Serial buses (USB, FireWire) are used for medium- to high-speed I/O connections. Interconnectivity of SCIs supports scalable multiprocessor clusters and high-performance modular real-time systems. Another version of the compact CS has evolved with a four-slot Micro PC chassis featuring a basic communication processor module, which also includes dynamic and static memory chips and a set of standard interfaces (CAN bus, RS232, and others). Real-time operating systems (RT-Linux, RTEMS) can be used for data acquisition and control applications. 2. Embedded modular real-time systems with passive backplanes, such as Euro-card (3U format), are international standards (IEEE 1101.1). The VME bus allows 16-bit data transmission in 3U format (6U boards support the full data bus bandwidth). Compared to VME (3U), cPCI (3U) is a higher-performance and more efficient system, while implementing PC functionality in the VME architecture is extremely difficult. The 3U cPCI bus outperforms the 3U VME. Compared to embedded PC board formats, the cPCI/PXI bus supports full 32-bit or 64-bit data transmission in both single-wide and double-wide boards. cPCI/PXI also offers several advantages. cPCI/PXI increases system flexibility, increasing the number of PCI slots from 4 to 8. cPCI is designed for industrial environments (such as VME), while PXI is designed for instrumentation systems (such as VXI). The 3U cPCI passive backplane is small but can be enlarged. Using a backplane makes maintaining and upgrading 3U cPCI modules much simpler. The cPCI/PXI (3U) board supports the I/O required for industrial automation, which also necessitates distributed I/O. cPCI supports fieldbus for data acquisition, control, monitoring, and process reporting. To meet the needs of industrial applications, the cPCI system supports advanced network connectivity features for cPCI single-board computers (10/100M Ethernet, USB, FireWire, and fieldbus). Modularity enables a wide range of applications and provides the flexibility of cPCI/PXI-based SBC support. Embedded modular cPCI/PXI (3U) system hardware offers the following advantages: 1) Its small form factor (220 pins, 2mm connector) makes it a robust platform resistant to shock and vibration in control applications. 2) A complete PC module (with graphics, Fast Ethernet, IEEE 1394, USB, fieldbus, flash memory, and 128M SDRAM) can be built on a compact and flexible 3U platform. 3) Reduced power consumption is a key step in cost reduction; smaller processor geometry lowers the power level. Studies show that control devices implemented via 3U cPCI typically consume less than 20W of power. 4) Additionally, 8-slot cPCI backplanes with a 64-bit bus offer economical backplanes (passive and active) through rack and EMI shielding accessories. Modern embedded computer solutions require Windows-based software for human-machine interface, network connectivity, file management, and deterministic real-time software for control applications (RT-Linux, RTEMS, QNX, OS-9, VxWork). Linux support for PC-compatible embedded SBCs tends to be provided in a normal, chip-based manner, including specific functions such as display controller modes, LCD panel control signals, PCMCIA, on-board solid-state drives, and non-standard functions (watchdog timers). 3. Distributed Systems Interconnected with SANs Because a large number of parallel processors in distributed data processing systems are limited by the bus, the Scalable One-Time Interface (SCI) has become the best System Area Network (SAN) for advanced multiprocessor architectures. Subsequently, the first high-performance modular multiprocessor system based on SCI with hardware consistency was developed. Based on multi-level physical models, it has been proposed to use a high-performance SAN architecture—an advanced integrated real-time system based on standard compact PCs (PC-boards) and link modules (such as Dolphin)—for high-performance data acquisition, control, and distributed data processing in experimental physics research. An optimal approach to building a high-performance real-time system is to use Industrial Computer Systems MB (ICS MB), PC MB, or cPCI/PXI, connecting them to the SAN via different topologies depending on the specific application. Distributed parallel data processing models include Symmetric Multiprocessing (SMP), Massively Parallel Processing (MPP), and cluster systems (RMC and NUMA). RMC (Mapped Memory Cluster) is a cluster system with memory replication and transfer mechanisms between nodes and communication connections. Leveraging the link modules of the System Area Network (SAN), the highly modular structure of the distributed integrated system supports efficient interaction between distributed processors and memory. The SAN comprises the following levels: 1) The core level consists of a set of core processors, memory, and I/O controllers, all interconnected. Compared to off-chip memory on the same board, new single-chip microcomputers have shorter communication links, easier access, and shorter data transfer times. 2) The atomic level (A-module) of the compact board structure of the system model includes processors for specific and general purposes. The simplest and most efficient real-time system for data acquisition and control can be based on a standard PC MB with a single-core, dual-core, or triple-core processor. There is a limit to the number of processor modules on the same bus. Symmetric multiprocessing (SMP) is the basic software model for multiprocessors. 3) The molecular level (macrostructure) depends on the system topology. A large number of multiprocessor nodes can be connected to large (thousand-processor) systems via SAN (“Big Bus” model) to support distributed integrated real-time systems for data acquisition, control, and data processing applications. 4) Interconnection of distributed systems is based on link, bridge, and switch modules (L-modules, B-modules, and S-modules). The cost of communication speed decreases much faster than the cost of pins and board space. Traditional communication is based on buses, which limits the number of processors. One viable solution is to use packet-based semaphore on many independent point-to-point connections, which can address the bus bottleneck but introduces new challenges—maintaining cache coherence in a shared memory model within the system. Weak interactions between processor modules are based on message passing (Ethernet). Medium-level interactions are based on external storage devices (disks, tapes) used in the cluster. Strong interactions between processor cores are based on direct reads of distributed memory, implemented on the SCI, which also supports weak interactions between processor modules. Strong interactions in the SCI include small packet transactions (packets with separate send and response sequences). Packet formats include write, read, move, and lock commands, where xx represents the allowed block length (number of data bytes, to the right of the packet header). Scalability is a concern for enhancing the performance of multiprocessor real-time systems (connected to systems with thousands of processors). The distributed memory model of the SAN architecture supports parallel pipelined data processing (computation) running as an SMP model within a single address space. 64-bit addresses support 256TB per node. Cache coherence supports data availability across all processors in a distributed parallel data processing real-time system. Real-time systems contain numerous processors that attempt to modify individual data or simultaneously cache backups of that data in their own caches. Consistency, implemented through software or hardware, prevents multiple processors from attempting to modify the same data at the same time. Hardware consistency supports high performance (high cost), while software consistency provides high performance (low cost). The topology of a modular real-time system should be developed based on a carefully selected set of modules to optimize the solution of deterministic problems. It should be a matrix for data acquisition with matrix detectors or a 3D topology for 3D imaging. In the control domain, the system should have a topology similar to a large machine (linear or ring-based). SAN-based distributed systems should share a 64-bit SCI address, with the high 16 bits used for forwarding data packets at appropriate nodes. The system topology can implement strong parallel pipeline interactions between processors based on simple rings, multiple rings, bridges, or switches. The SCI is based on point-to-point connections and supports transactions of all processor modules simultaneously. Commercial Dolphin's L module provides a bidirectional SCI connection of 800 Mbytes per second for moving large amounts of distributed data with small application potential (2.3 milliseconds) and reducing node control information with the best scalability for multi-point applications. The network-based distributed real-time system includes the following nodes: a controller (CS) connected to a virtual terminal (VS) to collect real-time data and output control data. A simple CS with one Ethernet port is based on a compact ICS MB with two PCI slots for data acquisition and control modules. Another port is used for expansion or additional Ethernet connections. The VS should support professional-grade simulation, monitoring, and testing. Virtual instruments and standard application software, based on basic operating systems (Windows, Linux), operate on VSs connected to a large number of distributed CSs running RTLinux via 10/100M Ethernet. Each VS should be capable of multi-server operation. A joint general model of a scalable modular real-time system integrates data acquisition, triggering, and control systems based on an interconnected network (Ethernet) and SAN (SCI). For engineering systems in the control technology field, a compact ICS MB (A module) with two PCI slots and Ethernet is a good platform. For high-performance data acquisition and triggering systems in the field of experimental physics, nodes based on embedded cPCI/PXI and interconnected with SAN (SCI) are a good platform.