Problem Statement Data integration is a core component of enterprise information systems, serving as a unified data platform to support other parts of the system. Many enterprises have or will have multiple business systems, each with its own data repository. The data from each business system is generally incomplete and inconsistent. Traditional solutions for data integration often require a centralized repository, which struggles to adapt flexibly to changes in the underlying data sources. Traditional solutions require direct data retrieval from the data repository, and only from the data repository. However, direct access to the data repository is often restricted, and the data in the repository is often raw data that requires processing by business logic to become valuable. Directly using raw data from the data repository is meaningless. Therefore, the key to data integration is how to easily obtain the required data and how to perform correct data integration. This paper proposes a real-time dynamic data integration service framework. This framework's data integration is dynamic, does not require a centralized repository, and is performed in real-time. The data source for integration is not limited to the data repository; it can be an application, a component, a service, or even the result of data integration as a new data source. Introduction to SOA Technology SOA, or Service-Oriented Architecture, can be viewed as a software system architecture. SOA primarily addresses the need for business integration in an Internet environment by connecting components with specific functions as service providers in a loosely coupled and unified interface definition manner to complete specific business processes. SOA has three fundamental characteristics: ■ Functional entities providing services in an SOA architecture possess completely independent capabilities, eliminating the need to concern oneself with their implementation methods and operational mechanisms; ■ SOA architecture accesses large amounts of data infrequently, aiming to exchange as much data as possible at a time during information exchange; ■ SOA architecture employs text-based rather than binary messaging, where messages themselves do not contain any processing logic or data types, and there is no need to concern oneself with the details of the message receiver. The maturity and widespread application of XML and Web Services standards have provided a foundation for the widespread implementation of SOA architecture. XML is a markup language designed for documents containing structured information. Using this description method, data exchange between applications can be performed while maintaining the original meaning and structure of the data, thus preserving the flexibility of data exchange between different systems. Web Services are based on the most widely accepted, open technology standards (such as HTTP, SMTP, XML, SOAP, WSDL, and UDDI). They support the separation of service interface description and service processing, centralized storage and publishing of service descriptions, automatic service discovery and dynamic binding, and service composition, becoming the infrastructure for building and integrating next-generation service-oriented application systems. Web Services can be defined as providing services over a network via the SOAP protocol, using WSDL to describe these services, and registering the services through UDDI so that users can find them. SOAP: This is the communication protocol for Web Services, using XML format to define messages, i.e., SOAP messages, which are well-structured XML segments contained within a pair of SOAP packets. Currently, it is often based on the HTTP protocol to transmit XML data. WSDL: Web Service Description Language. A WSDL file is also an XML document containing detailed descriptions of the Web Service, such as parameter types, function names, return types, and binding protocols. Callers can determine the interface functions of the Web Service by viewing the WSDL file. UDDI: This is the registry center for Web services. Web Service providers register their services with the UDDI registry center, and callers can then query this known UDDI registry center to find the Web services they need. Web Services providers implement service interface functions and service descriptions, and publish them to callers or register them with a service registry. Service callers select the required service by querying the service description locally or in the service registry to bind and call the Web Service interface functions. The service provider returns the service results to the service caller in the form of an XML document, completing the information exchange. Figure 1 shows the architecture of Web Services. [align=center] Figure 1 Web Services Architecture[/align] SOA-Based Data Integration The essence of data integration is to organically centralize data from different sources, formats, and characteristics, either logically or physically, thereby providing users with comprehensive data sharing. Many mature frameworks exist in the field of data integration. Currently, data warehouses and middleware-based approaches are commonly used to construct data integration services. The middleware pattern accesses heterogeneous databases, legacy systems, Web resources, etc., through a unified global data model. The middleware sits between various data sources and applications, coordinating downwards among the data sources and providing a unified data model and a common interface for data access to applications accessing integrated data. Applications from each data source still perform their tasks, while the middleware primarily focuses on providing a high-level retrieval service for various data sources. The middle layer provides a unified logical view of data to hide the details of the underlying data, allowing users to view the integrated data source as a single whole. The SOA-based dynamic data integration service framework proposed in this paper belongs to the middle layer pattern. In SOA-based data integration, XML provides a standardized data structure to assist in integrating different data structures between systems and presenting the integrated data in an associative view. "Communicating in the same language" is no longer necessary; Web Services will serve as a standardized communication method, allowing one part to dynamically discover the capabilities and needs of other parts. Therefore, integration is dynamic, and the data integration method can be organized as needed to obtain different integrated views; integration is also real-time, allowing for easy access to the latest data. SOA-based data integration transforms the traditional data integration solution's focus from how data is exchanged between different systems to how system functions are presented. Data is no longer obtained in a point-to-point manner but is a service freely available on the network. Systems do not interact at the underlying protocol level but exchange data at an abstract interface level. Systems merely present their functions as services, which other systems can easily discover and bind to at runtime or design time. The integrated services can be any application, system, and data source, regardless of their specific requirements. Research on Dynamic Data Integration Service Framework Framework Technology System Figure 2 shows the technical architecture of a SOA-based dynamic data integration service framework. As can be seen from the figure, the framework's technical system consists of five layers: data source adapter, service wrapper, SOA runtime engine, XML view engine, and XML view. [align=center]Figure 2: Framework Technical Architecture[/align] The data source adapter is the bridge for interacting with various data sources. It consists of pluggable components that are dynamically loaded. Data source adapters are written in a way that conforms to standard interfaces, allowing for independent development of adapters for new data sources and easy integration into the system. The type of data source is essentially unlimited; it can be a database, a text file located on the Internet, or even an application system. The data source adapter converts the raw data stored in various data sources into standard XML documents. The service wrapper wraps the data source adapter into a standard Web Services service, thus transforming the API access mode to the data source into a service provision mode. Service wrappers can wrap multiple data source adapters simultaneously. Another important function is that they can further process raw data from one or more data sources to transform it into more meaningful information, and can cache the processed data locally. This caching improves the efficiency and reliability of the entire system—the system can continue to operate even if a data source fails. Pre-processing business logic reduces the complexity of XML views, making them easier to understand. Service wrappers isolate the details of data access and the details of data processing logic; these two parts can vary independently, greatly improving the system's flexibility. The SOA runtime engine is the core of the framework, scheduling service execution and driving data integration processing. The SOA runtime engine includes rule processors, process processors, and message processors. The SOA runtime engine transforms the data integration requests parsed by the XML view engine into service invocation requests, and then finds, binds, and invokes the appropriate services. The XML view engine provides three mechanisms: First, it represents multiple heterogeneous data sources as a single, real-time virtual database, which consists of a series of real-time acquired XML documents. Secondly, it parses user requests from XML views, forwards these requests to lower layers, and returns the final result to the user in the form of an XML document. Finally, it expresses its metadata using XML Schema, which forms the basis of XML views, defining the XML view based on the XML Schema. An XML Schema is like a table in a relational database, while an XML view is like a view associated with a table in a relational database. The XML view engine itself is provided as a web service, allowing users to easily access it in a standard way. It can also serve as a data source for other XML view engines. An XML view is a metadata description of data integration, expressing the effect of the integration. It is not the result of integration, but merely metadata representation of the integration result, just like a view in a relational database, representing the compositional relationships that the actual data should have: which table the data comes from, what processing is needed for the table data, how data from different tables are combined, etc. Users can understand the data integration pattern expressed by the view by viewing the XML view. Different XML views can be defined according to different data integration needs; this is dynamic data integration—defining the integration method only when needed, and obtaining the actual data only at runtime. As shown in Figure 2, these five layers can be divided into three parts, each developed and deployed independently in a distributed manner. The XML view and XML view engine form one part, the SOA runtime engine another, and the service wrapper and data source adapter yet another. This distributed data integration offers high flexibility, with the three parts operating independently. Similar to the service wrapper, multiple XML view engines can be deployed simultaneously, and each XML view engine can be encapsulated by a service wrapper into a new data source. This mechanism can easily meet the complex environments of modern enterprise data integration, achieving flexible data integration requirements at a relatively low cost. Figure 3 shows a typical deployment scheme. In Figure 3, XML view engine B uses service wrapper c, while service wrapper b wraps XML view engine B into a new data service. XML view engine A uses service wrapper a and service wrapper b, meaning it uses a certain XML view from both service wrapper a and view engine B. [align=center]Figure 3: Typical Deployment Scheme of the Framework[/align] The framework's operating mechanism is as follows: First, the user views the XML view to obtain the metastructure of data integration on that view, and initiates an appropriate processing request based on this metastructure. The XML view engine parses the request according to the corresponding XML view description in the processing request and forwards it to the SOA runtime engine. The SOA runtime engine transforms these requests into calls to the service wrapper. The service wrapper then transforms them into calls to the data source adapter, which performs the actual data source access. The data retrieved from the data source is processed by the service wrapper and transformed into a standard XML document, which is then sent back to the XML view engine. In the XML view engine, the actual data integration is completed according to the integration method description of the XML view, and the integration result is returned to the client. As can be seen from the framework's operating mechanism, data integration is real-time, and each request needs to retrieve data from the data source immediately. The basic operating mechanism of the framework is shown in Figure 4. [align=center]Figure 4: Framework Operation Mechanism[/align] Framework Structure In the SOA-based data integration framework, its service-oriented architecture determines the relationships between the modules within the framework: these modules can all be considered services from an SOA perspective, and all these services are loosely coupled together. The core part of the framework, the SOA runtime engine, primarily manages these various services, coordinates the interactions between services, and allows users to easily access these services. Figure 5 shows the overall structure of the framework. [align=center]Figure 5: Overall Framework Structure[/align] In Figure 5, the built-in services are the essential service functions for the normal operation of the framework. These services are used at the framework's underlying level and are invisible to users. The built-in services mainly include: a process engine, which provides the framework with process customization and execution functions; a rules engine, which defines the integration and processing methods based on data integration rules, allowing enterprises to define integration and processing methods according to their own business rules; a service registration and discovery service, providing a registry center and service discovery function for Web Services; and a message processor, which provides a message processing mechanism so that services can interact via messages. Plug-in services are the main mechanism that enables the framework to have high scalability and flexibility to adapt to various needs. It packages various data sources into services and embeds them into the framework in a pluggable manner. Plugin services do not directly access data sources but access them through data source adapters, thus decoupling access to data sources from service provision. Data source adapters can be dynamically assembled at runtime as needed. Simultaneously, plugin services can preprocess the data obtained from their data source adapters as needed to reduce the complexity of final integration and improve the overall system efficiency. Plugin services can extend the entire system with minimal programming or simply configuration, offering great flexibility. Proxy services are an extended type of plugin service that provides wrapping for data sources such as third-party applications (ERP, CRM, etc.) and XML view engines. The main difference between proxy services and plugin services in data source wrapping is that plugin services wrap data sources with explicit API interfaces, such as databases, EJB, COM, MOM, and SOAP, while proxy services wrap data sources without explicit API interfaces, requiring more processing. Proxy services also serve as a mechanism for recursive data integration, using the integrated data as the data source for other data integrations. The Service Manager manages all services within the framework and serves as the entry point for service access. It locates the correct service, manages the service lifecycle, monitors service operation, schedules service execution, and registers services. The Service Factory is primarily used to generate various services. When the Service Manager searches for a service and finds it not yet running, it requests the Service Factory to dynamically generate and run the service. The Service Factory starts the service based on its registration information. If the underlying data source for the service is unavailable, the service will attempt to load locally cached data. The Access Control Module provides the foundation for secure service use and data access. Besides providing role-based access control, it has two other important functions: First, it employs a tag-based user authentication model, allowing a single login to access multiple data sources, greatly simplifying user access patterns. Second, the Access Control Module also handles the lifecycle of user access sessions, ensuring users only access the system for a specific period, further enhancing system security. The Service Access Channel is the channel through which users access services wrapped in an XML view engine and parses user requests into a series of query plans that directly access the underlying services. Query plans are cached to maximize their reuse. When a new request arrives, the service access channel compares it to an existing query plan; if one is available, it is used directly, significantly improving system efficiency. A single service access channel can serve multiple users simultaneously. The access channel management module provides runtime management of the service access channel. It can merge requests that might return the same data into a single request, prioritize multiple user requests to ensure high-priority requests are processed quickly, or multiplex small-data requests from multiple users onto a single service access channel to ensure sufficient access for large-data requests. These mechanisms reduce system resource consumption and improve operational efficiency. The service publish/subscribe interface provides an asynchronous communication mechanism within the framework. Various services can subscribe to desired messages through the service publish/subscribe interface. When a suitable message is published, all subscribed services are notified—a many-to-many communication mechanism. The service publish/subscribe interface can also connect to various message middleware to provide message-based service interconnection functionality. Meanwhile, through service publish/subscribe interfaces, the framework will possess certain transaction processing capabilities. The SOA-based dynamic data integration service framework provides a service-oriented architecture framework for the integration and sharing of heterogeneous data. Based on XML technology and using dynamic integration as a means, it achieves data integration between various application systems, realizes cross-platform data sharing, and achieves the goal of cross-platform data resource integration, realizing information interconnection. The SOA-based dynamic data integration service framework can provide effective solutions and approaches for addressing technical challenges such as heterogeneous interfaces, real-time data exchange, and on-demand responsiveness in cross-platform data sharing and integration.