Abstract : This paper presents a novel intelligent elevator monitoring system based on intelligent video surveillance, capable of identifying unsafe conditions in elevators and abnormal behavior of passengers inside the elevator car. The overall system design and framework diagram are provided. The system hardware platform is constructed with video acquisition, transmission, and processing modules, power supply, CPU, communication, and control modules, as well as three types of servers as its main functional components, and detailed hardware selection is given. Based on C++ and OpenCV programming, combined with UDP communication, serial communication network technology, and database technology, the paper elaborates on the overall software structure, functional requirements, monitoring algorithm design, and the judgment workflow for abnormal human behavior, elevator malfunctions, and passenger count.
Keywords : elevator; video; Internet of Things; intelligent monitoring system
0 Introduction
By the end of 2013, my country had recorded 70 elevator accidents, accounting for 30.82% of all special equipment accidents and 19.72% of all fatalities during the same period. The safe operation of elevators has garnered widespread public attention. However, significant shortcomings exist in their use, supervision, maintenance, and security, particularly in complex public spaces. There is an urgent need to establish a universal, remote, real-time monitoring system for unsafe elevator conditions and abnormal human behavior.
In this context, based on actual engineering needs, a comprehensive monitoring/control platform with a central server is built using intelligent video surveillance (IVS), IoT, video image digital processing technology, and other information technology. This platform will feature functions such as human feature extraction, abnormal behavior detection, people counting, elevator operation data monitoring, fault monitoring, remote active control, automatic alarm, and active rescue. This will reduce the incidence of elevator malfunctions and abnormal behavior-related personal safety accidents, improve processing efficiency, enable remote control, and eliminate potential safety hazards to people and objects in their early stages. Furthermore, it will break the monopoly of foreign manufacturers on networked monitoring systems based on their own products, promoting the complete localization of monitoring systems.
1. Overview of System Overall Design
IVS elevator intelligent monitoring technology is an extension and development of the original control system. It mainly includes two parts: all-weather remote monitoring and control of unsafe conditions of objects and abnormal human behavior (referred to as object module and human module). Its core is a B/S architecture information management platform in the monitoring center, which monitors the operational information and real-time video images of people inside each elevator from independently installed sensors in real time. Through precise combined algorithm analysis, it identifies unsafe conditions and abnormal human behavior, providing real-time alarms and early warnings, and proactively taking corresponding protective measures to minimize the degree of harm. On one hand, it utilizes prior knowledge to build behavioral models, using Laplace decomposition and adaptive Gaussian mixture models for background modeling. This enables real-time video monitoring based on foreground edge processing, human feature extraction, and feature trajectory recognition technologies. A composite algorithm is used to intelligently identify abnormal behaviors such as sudden falls, robberies, and violent attacks, and to accurately count the number of people inside the elevator. On the other hand, it predicts and eliminates potential faults, and effectively handles faults at the first sign of trouble, achieving typical fault monitoring and proactive alarm and rescue functions for trapped passengers. It also features RFID radio frequency identification reading/writing/maintenance management for maintenance personnel; and establishes a database to store data streams for querying, statistical analysis, and basic information about elevators, user units, and elevator industry enterprises, better serving society and the public.
2. Monitoring System Design
Using the RS-485/232 standard, a dedicated line was laid, and server and front-end monitoring software were designed and developed. The monitoring center computer remotely monitors elevators in various locations. The communication protocol conversion was extended to unify the information codes of different elevator controllers into standard information codes, enabling centralized monitoring of elevators of different manufacturers and models. Input terminals were set up when the data acquisition device monitored various operating status signals and fault signals to improve fault detection capabilities and realize fault diagnosis and fault prediction [1-3].
The elevator system module mainly consists of a data acquisition unit (i.e., a remote monitor) that interfaces with the elevator via a CAN bus or inputs signal status data; a network data converter that converts RS-485 or RS-422 data into TCP/IP data; a configuration terminal with configuration software installed to configure the network data converter; and a monitoring server with monitoring software installed. For elevators in complex public places, a microcomputer control system with high scalability, flexibility, safety, and stability is selected to meet various needs; additional interface circuits are added to achieve direct control of 110V or 220V electrical equipment.
The human system module adopts the MPEG Moving Picture Experts Group video coding standard under ISO, and uses HTTP/TCP protocol to transmit control information and RTP/UDP protocol to transmit real-time data streaming media transmission technology. The module mainly consists of an in-car monitoring terminal, a property central control room, a monitoring center, and other remote control terminals. An integrated PTZ camera is used to collect in-car video signals in real time. The signals are sent to a DVS-7000 network video server (implemented with a DSP chip) via an RJ-45 serial cable for real-time compression and encoding of the video images to restore the video. The data is then transmitted to the data forwarding center in the property central control room via wired or wireless network. The center calls a composite algorithm that combines human shape analysis and visual tracking to locate the human body in the car and segment the head, hands, torso, and feet to establish a human appearance model, identify various abnormal behaviors, and actively alarm and send corresponding countermeasure commands. The video data is stored in a local area network C/S architecture database server and a WEB server through C# language with embedded SQL programming. Data is sent through a B/S mode, and the monitoring center and other remote clients can monitor the operation status through a browser.
2.1 System Structure
The system development achieves generalization, modularization, informatization, and intelligence to ensure the normal implementation of the functions of each system and its subsystems, and to guarantee the needs of sustainable development. Each program is highly modularized and has an extensible technical framework and standard external interfaces, reserving interfaces for application systems outside the system and secondary development. The platform architecture of the system implementation [4-5] is shown in Figure 1.
The system mainly consists of a monitoring center, a communication system, on-site data/image acquisition, and communication devices. 1) Monitoring Center: This is the control and management center, responsible for issuing monitoring and communication commands to each monitoring front-end, collecting and processing monitoring data, supervising or directing the operation of equipment, and providing information services to the front-end through a data publishing system. 2) Communication System: The communication system includes wired communication lines and communication software. The data communication system supports RS-232/RS-485 data transmission protocols. 3) On-site Data Acquisition and Communication Devices: CAN-BA acquires elevator data, BN-030QA/SE-C acquires images, and relevant data is transmitted via RS-232/RS-485 communication. The system modules specifically include: a data acquisition and transmission module, a system security module, a central control module, a database platform module, a comprehensive information management platform, and an operation and maintenance module.
Figure 1. Framework diagram of the elevator object/person intelligent monitoring system
2.2 System Communication Network
In complex public places, elevator safety is paramount, data transmission volume is large, real-time performance is high, the environment is complex, and there is strong magnetic interference. The initial choice is a wired network monitoring system with strong real-time performance and high reliability. On this basis, GPRS wireless communication is extended. The data transmission of the monitoring system is characterized by small flow, many scattered points, discontinuity[2], frequency and suddenness. A wireless network monitoring system should be adopted. Each elevator needs to be equipped with a GPRS data transmission terminal, which is responsible for exchanging data with the monitoring unit and controlling the internal communication between the monitoring center and the car. Dual-line configuration can be used as needed, and the mode can be selected by the user. In the future, wireless communication should be the main method and wired communication should be the auxiliary method.
2.3 System Functions
Without altering the original elevator structure, interfering with the control loop, or affecting normal operation, a set of sensors and video imaging systems, independent of the elevator's electromechanical system, are used to monitor necessary status information and abnormal behavior of people inside and outside the car during elevator operation. The monitoring system includes key functions such as dynamic display of object/person status, human feature extraction, abnormal behavior detection, people counting, status transitions and fault code conversion, full-duplex voice intercom for car occupants, fault status judgment, automatic telephone/SMS alarm, active audible and visual alarms for entrapment/abnormal behavior, video/image/voice reassurance, maintenance attendance records, maintenance information statistics and management, backup power switching for detection terminals, status parameter storage, archive information, annual inspection information statistics and management, proactive and targeted elevator control, and real-time video monitoring.
3. Monitoring System Hardware Design
3.1 Elevator Unsafe Condition Monitoring Module
The core hardware consists of four modules: power supply, CPU, communication, and control. The power supply module provides power to the CAN-BA. The CPU, the control center, sends the collected data streams to the ThinkServer TD340 server and processes user commands from the host computer. The communication module's CAN communication converts the collected differential signals into differential-mode signals recognizable by the CPU, while MAX232/485 communication handles data transmission. The control module mainly includes an external detection module and a relay output control module. The external detection module collects real-time operating information, while the relay output control module, upon receiving instructions or experiencing abnormal vibrations such as earthquakes, promptly stops the elevator at the nearest floor to ensure passenger safety before halting operation. The relationships between these modules are shown in Figure 2.
Figure 2 Hardware Module Relationship Diagram
The power supply module, considering factors such as heat dissipation, low power consumption, low ripple, cost, interface voltage withstand, linear regulation, pin voltage matching, and stability, selects the switching LM2575. The CPU chosen is the low-power, high-performance AT89S52 CMOS 8-bit microcontroller, compatible with the 80C51 pinout and standard MCS-51 instruction set, providing a flexible and effective solution for embedded control applications. The communication module mainly includes: the AT89S52 responsible for SJA1000 initialization and data transmission/reception; the SJA1000 independent CAN bus controller for local area network control; the TJA1050 responsible for converting differential signals into differential-mode signals for subsequent CPU operations; and a CAN communication circuit using a 6N137 opto-isolator to ensure communication is free from noise interference. To prevent external interference, a communication isolation module is designed to filter the signal, verify continuity with a multimeter, and account for voltage drop. Serial communication is achieved through a MAX232 communication and an optocoupler isolation circuit composed of the 6N137 opto-isolator. The system has insufficient driving capability and poor communication quality for long-distance data transmission. Therefore, it is necessary to convert the TTL level to the appropriate level of EAI through MAX232/485 to realize communication between the microcontroller and the PC. The analog circuit and the AD converter and digital circuit are completely isolated by 6N137 to improve the key parameter of signal-to-noise ratio and suppress the interference of noise on the analog circuit and AD circuit. It should be noted in the design that 6N137 must be connected in series with a current limiting resistor and a pull-up resistor [5]. The control module consists of three parts [4]: TLP620-4 dual optocoupler conduction real-time detection of door opening and closing, vibration, fire protection, overload and other signals; use optocoupler isolation to prevent excessive current and use light-emitting diode to indicate; select parallel voltage limiting transient voltage protection diode TVS working circuit to control the surge voltage within the safe value and discharge through ground wire.
3.2 Personnel Abnormal Behavior Monitoring Module
The BN-030QA/SE-C PTZ integrated camera was selected; the BN-80013 intelligent decoder, featuring strong anti-crash performance, support for DC12V/AC24V output, lens motor protection, and automatic matching control protocol, was chosen; the data was transmitted to an MPEG-4 DVS-7000 video server for encoding, compression, and transmission; the Lenovo ThinkServer TD340 in the property monitoring room's data forwarding center received the acquired video data via antenna, and the forwarding center supported and stored the data on the TD340 using software; the background image was then extracted using a combination of complex algorithms. The system can model human and scene limbs and identify abnormal behaviors to proactively respond; it also supports the Advantech IPC-610H industrial control computer to observe video images, play back historical videos, and manually control the pan-tilt-zoom (PTZ) direction; it uses HTTP/TCP protocol to transmit control information and RTP/UDP protocol to transmit real-time data. The monitoring center can enter a URL through the client, and according to the HTTP protocol, the Lenovo R680G7 Web server will send data query, query optimization, multi-user storage control, update, and data manipulation requests to the Lenovo ThinkServer T350 database server in the local area network, and return the response results to the client.
4. Elevator IoT Monitoring System Software Design
Designing a system with good real-time human-computer interaction is particularly important. Java is slow and has a large runtime environment; Visual C++ 6.0 has poor template support, is prone to freezing during compilation, and only supports the Windows operating system, resulting in poor compatibility and the possibility of cpp files failing to open. Therefore, C++ was ultimately chosen, and OpenCV, an open-source software that is highly portable, supports cross-platform construction, does not rely on virtual mechanisms, allows the direct use of many algorithms, and boasts powerful performance and speed advantages, was loaded to build the intelligent monitoring system software platform for objects/people.
4.1 Software Functional Requirements Analysis
The software structure functional modules are divided into a display layer, a data processing layer, a data transceiver layer and an auxiliary module. The display layer is used to display the parsed data from the data processing layer and receive user interaction commands and transmit them to the lower layer. It includes monitoring interface, user interaction, user action explanation and data display. The data processing layer is used to parse the data from the data transceiver layer and save abnormal data. At the same time, it sends the parsed data to the display layer as display parameters and provides database read/write driver, data configuration, analysis and other data processing functions. It includes data parsing, data transmission, database read/write driver, data saving, data statistical analysis, data configuration, data verification and so on. The data transceiver layer is used to obtain the monitoring data of the collector and also send the data from the data processing layer to the collector. It includes interface type configuration, data read/write and data verification. The auxiliary module [1] includes functions such as single elevator display, data export, information configuration, information query, icon setting, icon sorting, elevator number setting, software main interface setting, communication serial port, control relay, warning window, initialization window and so on.
4.2 Software Workflow
The specific process is as follows after system startup: 1. Call the user management module and select the login method; 2. Initialize the program to obtain the elevator number, interface type, video capture window, port number, and network IP; 3. Determine the data transmission method, initialize the serial port if necessary, set parameters, and capture the current video frame data information; 4. Set the ROI open interface, confirm the serial/network port to send C180FF, and the elevator replies with a data stream; 5. Perform subtraction and extract the leading edge background, and determine whether a data stream has been received. If not, use a heartbeat algorithm to determine online status. If offline, notify personnel for maintenance; if online, resend the query data; 6. Apply sharpening and noise reduction measures to the background edges. The system performs the following steps: 1. It calls the algorithm to build and process a suitable model, while simultaneously reading serial/network port data and determining if the first byte is the correct elevator number. If incorrect, it re-enters C180FF. 2. It correctly verifies the data, checking if the sum of the data is equal to FFH and if it exceeds the set energy and passenger thresholds. 3. It parses the elevator and human body model data, as well as the passenger count data, and saves it to the software log. 4. It checks for faults and abnormalities. If any are found, the information is displayed in the audible and visual alarm window, and personnel are notified for maintenance. If no faults or abnormalities are found, the parsed data can be transmitted to the display layer as display parameters. Finally, the display layer displays the parsed data from the data processing layer.
The data stream sent back from the elevator is transmitted to the display layer for display, providing users with intuitive and clear feedback on elevator information. Simultaneously, users can perform relevant elevator operations. First, the user commands are parsed, then the data is sent to the lower-level computer. The CAN-BA data collector's CPU processes the data to control the elevator for correct operation.
5 Conclusion
Based on in-depth research into complex public spaces and elevator monitoring systems, this paper analyzes and designs functionalities, determines the overall system design scheme by considering communication and structure, and designs the system architecture and logical flow. The system hardware is designed according to two main themes: unsafe conditions of objects and abnormal human behavior, and a system operation platform is built. A simplified system structure diagram and detailed selection of relevant hardware are provided. Based on a combination of B/S and Qt, the monitoring system application software is modularly designed, and a detailed workflow for the system's detection and analysis of video images, abnormal behavior, and unsafe elevator conditions is designed. Preliminary laboratory functional and performance tests have been completed. The next main tasks are to improve the abnormal behavior detection module, install and debug on-site equipment, conduct system trial operation tests, and perform system integration and operation.
About the author: Zhang Jinyang (1976-), female, master's degree, senior engineer.
Author's Affiliation: Xinjiang Uygur Autonomous Region Special Equipment Inspection and Research Institute
