The research focus of XAI (Explainable AI) is ensuring that the reasoning and decision-making of AI systems can be understood by human users. In the military field, explainability typically guarantees the following: AI systems operated by human users possess appropriate mental models; experts can gain insights and knowledge from AI systems and their implicit tactical and strategic behaviors; AI systems comply with international and national laws; and developers can identify defects or errors in AI systems before deployment. Based on the Swedish Defence Research Institute report, "Exploring Explainable Artificial Intelligence Technologies in Military Deep Learning Applications," this article argues that such AI systems are inherently difficult to understand due to the complexity of their modeling processes, which prevent the use of alternative explainable methods. Although the field of XAI in deep learning is still in its early stages, several interpretable techniques have emerged. Current XAI technologies are primarily used for development purposes, such as error identification. However, if these technologies can create appropriate mental models for user-operated AI systems, facilitate tactical development, and ensure that future military AI systems comply with national and international laws, then further research is warranted. This article will introduce XAI technologies and their applications in the military, based on the report.
Keywords: Artificial Intelligence, Explainable Artificial Intelligence, Deep Learning. The main reason for the success of artificial intelligence (AI) today is the breakthrough in machine learning (ML), more precisely, the breakthrough in deep learning (DL). Deep learning is a technology with disruptive potential; using deep neural networks, people can achieve complex modeling that traditional technologies cannot. For example, deep learning can be used for accurate transcription (speech to text), translation (text to text), playing real-time strategy games (image to action), lip reading (image to text), facial recognition (image to recognition), and controlling autonomous vehicles (image to action), etc.
However, since deep learning is still in its early stages of development and there is no mathematical framework that can guarantee the accuracy of the model, many challenges and problems will inevitably be encountered when developing, deploying, using and maintaining military neural network models, requiring people to constantly think and find solutions.
For military personnel, including combatants and data analysts, the greatest challenge may lie in explainability. Experience shows that the need for explainability increases significantly when actions impact human life. Explainability is crucial because it influences users' trust and reliance on the system. A balance must be maintained: excessive trust leads to misuse of the system, while insufficient trust renders it ineffective. Ultimately, explanation aims to help users build appropriate mental models of the system to ensure its effective utilization.
Deep learning has the potential to enhance the autonomy of complex military systems such as fighter jets, submarines, drones, and satellite surveillance systems, but it can also make these systems more complex and harder to interpret. The main reason is that deep learning is an "end-to-end" machine learning technique, meaning that machines achieve high performance by learning to extract the most important features from input data. This process, distinct from traditional techniques that extract features intuitively by humans, is called representation learning. Representation learning often delivers high performance, but it also requires models to be highly expressive and non-linear. Therefore, deep neural networks trained using deep learning can contain millions or even billions of parameters, making these models difficult to interpret even with a deep understanding of the algorithms, model architecture, and training data.
In 2016, the U.S. Defense Advanced Research Projects Agency (DARPA) launched the Explainable Artificial Intelligence (XAI) project, with the aim of: 1. generating more interpretable models while maintaining high levels of learning performance (predictive accuracy); and 2. enabling human users to understand, appropriately trust, and effectively manage next-generation AI tools. Since its inception, the project has achieved several technological advancements. Some XAI technologies have been packaged into software libraries and are operational. Military personnel can utilize these libraries to gain deeper insights into deep neural networks, while simultaneously eliminating and validating their errors. This step is correct in its general direction, but from a military perspective, tailoring XAI technologies and tools to military users is equally crucial, requiring a high level of interpretability.
XAI technology
XAI is a key component in any high-risk military decision-making AI system that impacts human life. For example, in tactical-level AI applications focused on short-term decision-making, the capabilities include autonomous control of unmanned vehicles and target identification, tracking, and engagement of weapons and surveillance systems. XAI is equally important at the operational and strategic levels of warfare, where long-term decision-making and planning activities can affect all of humanity. At the operational and strategic levels, AI systems are typically used for information analysis and also for proposing plans or course of action through simulations. The main roles of XAI in military systems include:
Mental Modeling: XAI allows users to create suitable mental models for AI systems. Regardless of whether a military system utilizes AI, users must have a clear understanding of the system's operational boundaries to ensure its rational and effective use.
Insight: Deep neural networks can be used to acquire knowledge and identify models unknown to humans in complex processes. By using XAI technology, this knowledge can be discovered and learned. Developing tactics and strategies using reinforcement learning is a typical application of XAI. During development, XAI may generate deeper insights into the military field.
Legal and regulatory frameworks: XAI can be used to ensure that AI systems comply with national and international laws. Lethal Autonomous Weapon Systems (LAWS) are perhaps the most controversial AI application. Some want to ban such applications, while others argue that LAWS can exist as long as it improves accuracy and minimizes collateral damage. A report from the Swedish Institute of Defence Studies suggests that XAI can play a significant role in developing rules governing the timing and location of the activation of AI systems like LAWS.
Error Elimination: Numerous case studies in the literature demonstrate the use of XAI to identify errors in deep neural networks. Typically, deep neural networks will malfunction if copyright watermarks in images, fake simulator data, or unrealistic game data appear in the training data. They may perform well on test data but fail frequently on real data. Integrating XAI technology into the development process allows these problems to be detected and resolved before deployment.
XAI technologies mainly include: global interpretation techniques, such as visualization techniques for large high-dimensional datasets and model evaluation; local interpretation techniques, such as gradient significance, correlation score propagation techniques, Shapley value annealing interpretation, locally understandable model-independent interpretations, and random input sampling for interpreting black-box models; and hybrid interpretation techniques, such as spectral correlation analysis.
Evaluation of XAI technology
A frequently overlooked yet crucial aspect of XAI (Xonxia AI) is the evaluation of proposed XAI techniques. This section will introduce evaluation criteria starting from the human factor perspective. In human factor evaluation, users such as combat personnel and analysts are central to measuring the effectiveness of XAI in AI systems. This section will also introduce testing methods that can be used to compare localized XAI techniques.
1. Assessment of human factors
Human factor evaluation of XAI technologies will test whether each explanation takes into account all important factors so that users can fully utilize the AI system. For example, users may have different purposes, needs, knowledge, experience, task context, use cases, etc. As with the development of various systems, it is crucial to consider these factors throughout the entire AI system development process, from system specifications to user testing. Because deep learning XAI technology is an emerging research field, its initial users are typically system developers interested in evaluating model performance. However, it is currently uncertain whether these XAI technologies will be useful to military users. The article "Explainable AI Metrics: Challenges and Prospects" provides six metrics for evaluating explanations:
Explanation Quality: During the development of XAI technology, a checklist was compiled from the user's perspective. This checklist is based on existing literature on explanation and evaluates explanations from seven aspects, such as whether the explanation helps users understand how the AI system works, whether the explanation satisfies the user, and whether the explanation is detailed and comprehensive enough.
Explanation Satisfaction: A measurement scale that measures users' experience with explanations based on their quality. The scale includes eight items presented in declarative form (seven quality items and one item regarding whether the explanation is useful to the user's goals). A validity analysis showed that the scale is highly reliable and can be used to distinguish between good and bad explanations.
Mental Model Guidance: A good explanation can deepen users' understanding of how AI systems work and their decision-making principles. In cognitive psychology, this is called the user mental model of the AI system. This article suggests using four tasks to measure the user mental model of an AI system, such as the cue-based review task, where users describe their reasoning process after using the AI system to complete a task; and the prediction task, where users predict the subsequent behavior of the AI system. A study comparing user mental models and expert mental models demonstrates the completeness of the user mental model.
Curiosity-Driven Level: Good explanations can drive user research and address knowledge gaps in mental models. This article suggests measuring curiosity-driven level by asking clients to identify the factors that prompt them to seek explanations. Possible drivers include the rationality of the AI system's actions, reasons for excluding other options, and reasons why the AI system's operation does not meet expectations.
Explaining Trust Level: A good mental model allows users to maintain a moderate level of trust in the AI system and operate within its operational limits. The article suggests using a measurement scale covering eight items to assess user trust in the AI system. These items include user confidence in using the system, the system's predictability, and reliability.
System Performance: The ultimate goal of XAI is to improve the overall performance of the system, making it superior to an AI system without XAI technology. Performance metrics include the completion rate of main task objectives, the predictive ability of the AI system to respond to user feedback, and user acceptance.
Future research will explore how to understand these metrics when evaluating XAI technologies for AI systems.
2. Evaluation of local interpretation techniques
The visual effect of a saliency map varies depending on the data type the model processes. For example, heatmaps are typically used for image processing, while color-coded characters and words are typically used for text processing. Figure 1 illustrates the visual effect of creating a saliency map using heatmaps. This case uses gradient saliency (1.b) and the relevance score propagation technique (1.c) to generate a heatmap for the digit 0 (1.a). Important dimensions such as pixels in the image are represented by warm colors such as red, orange, and yellow, while less important dimensions are represented by cool colors such as dark blue, blue, and light blue. The significant differences between the two techniques are visually demonstrated by the location of the highlighted dimensions. This section will continue to introduce techniques for quantitatively comparing and evaluating local explanations to identify those that provide the most accurate interpretation.
Figure 1. MNIST images and their corresponding heatmaps; the heatmaps were generated using a gradient significance and correlation score propagation technique. Important dimensions or pixels in the figures are represented by warm colors such as red, orange, and yellow.
①Delete
During the process of altering or deleting inputs, a deletion metric can be calculated by measuring the model's accurate predictive ability. It's important to note that in this context, deletion means transforming the input values into something neutral, such as an image background. The deletion process is guided by a saliency map generated by XAI technology, ensuring that values in relatively more important dimensions are deleted before those in less important dimensions. During deletion, performance degrades rapidly if the interpretation is better, and slowly if it is worse.
Figure 2 uses the gradient saliency plot from Figure 1.b to illustrate the deletion process. In Figure 2.b, 50 of the most important pixels were deleted; at this point, it is still easy to see that the image represents the number 0. In Figure 2.f, more than half of the pixels (400 pixels) were deleted; at this point, it is difficult for people to recognize that the image represents the number 0.
Figure 2. Six images exported from the MNIST image deletion process, with 0, 50, 100, 200, 300, and 400 pixels deleted respectively.
② Insert
Insertion metrics are a complementary approach to deletion. Figure 3 illustrates the insertion process using MNIST images used during deletion. The all-black image in Figure 3.a represents the initial input; as more and more input dimensions are inserted according to the priority order of the saliency map, an improvement in accuracy can be detected. During insertion, the more information inserted into the input, the higher the model's prediction accuracy should be; that is, with better interpretation, accuracy improves faster, and vice versa.
Figure 3. Six images exported from the MNIST image insertion process, with insertions of 0, 50, 100, 200, 300, and 400 pixels respectively.
③ Evaluation indicators
This report demonstrates the deletion and insertion process using gradient significance and relevance score propagation techniques. A classifier and 100 randomly sampled images from the MINST dataset are used in the demonstration to evaluate the XAI technique.
Figures 4 and 5 illustrate the results of the insertion and deletion processes, respectively. The area under the curve (AUC) is a measure that can be used to quantitatively compare multiple XAI techniques. In the deletion process, a smaller AUC value is better than a larger AUC value, while in the insertion process, the opposite is true, with a larger AUC value better than a smaller AUC value.
As shown in Figure 4, the performance curve of the correlation score propagation technique shows a larger decrease and converges to a lower average probability value during the deletion process. This is consistent with the heatmap, which shows fewer warm colors compared to the gradient significance heatmap (see Figures 1.b and 1.c). This indicates that the correlation score propagation technique can find explanations faster with fewer features compared to gradient significance. The same conclusion can be drawn from Figure 5. As shown in Figure 5.b, the average probability rises rapidly after inserting only a few dozen features and reaches a high performance level after inserting about 100 features.
Figure 4. Deletion curves of gradient significance and correlation score propagation techniques.
Figure 5. Insertion curves of gradient significance and correlation score propagation techniques.
in conclusion
Deep learning will be used to supplement and replace some functions in military systems. In fact, military surveillance systems designed to autonomously detect and track targets of interest from massive amounts of image data have begun evaluating deep learning technologies. Compared to traditional software technologies, deep learning offers several advantages, the most important of which is its ability to handle complex modeling processes that are impossible with traditional software. Furthermore, deep learning can facilitate active learning, enabling the acquisition of high-quality data to enhance operational system models through interaction between AI systems and users.
However, these advantages also bring challenges at the technical and operational levels. The report focuses on the challenges posed by interpretability. The disadvantage of deep learning is that, while the learning algorithms, model architectures, and training data are not unfamiliar and are easy to understand, the behavior of the model itself is difficult to explain. This is usually not a problem in civilian applications such as music recommendations and ad suggestions, but in the military field, understanding and explaining the behavior of AI systems is crucial. This is because, whether at the operational level or at the strategic level where long-term decisions require military leaders and political decision-makers, the decisions and recommendations provided by AI systems can have a profound impact on the lives of all humanity.
While complex military systems such as fighter jets, submarines, tanks, and command and control decision support tools are equally difficult to master, the technologies that build these systems are inherently interpretable. Therefore, if errors occur in these systems, the problem can be identified and resolved by examining the entire system. However, this is difficult to achieve in the field of deep learning. Real-world deep neural networks typically consist of millions or even billions of parameters, making it impossible even for the model's creators to systematically address errors within the model.
The report proposes several cutting-edge XAI technologies to address the interpretability challenge. It is worth noting that while this report represents some progress in this area, AI technologies for military deep learning applications are still in their early stages of development. Furthermore, the XAI technologies proposed in this report have not yet been tested in military environments, therefore there is no guarantee that existing XAI technologies can enable high-risk military AI systems to utilize deep learning.
